Working with Streams

You've learned that local media captures the audio and video data of the current user, and remote media displays the audio and video data of other users. What's needed is some way to tie these together. The object that does this is known as a stream. A stream is a relationship between two end-points that determines how data will be sent and received. This section will focus on how to use send and receive data by connecting the LocalCameraMedia and RemoteMedia classes using streams.

Creating Bi-directional Streams

The most common type of stream is one that both sends and receives data. This is known as a bi-directional stream, because media data moves in two directions. To stream both audio and video data, you must create two streams, one for audio data and one for video data. The FM.IceLink.AudioStream class controls the transfer of audio data and the FM.IceLink.VideoStream controls the transfer of video data. To create bi-directional audio and video streams, create an instance of each class, and specify your LocalCameraMedia instance and a RemoteMedia instance as the parameters.

To review, you will only ever have one LocalCameraMedia instance but you will have multiple RemoteMedia instances - one for every participant in the session. Each stream defines how data flows between two end-points, which means that you will need one set of streams for every participant in the session. For each RemoteMedia instance that you create, you will also create an AudioStream and a VideoStream, which links the local and remote end-points together.

The code below shows how you would create a single set of streams:

var localMedia = new LocalCameraMedia();
var remoteMedia = new RemoteMedia();

var audioStream = new FM.IceLink.AudioStream(localMedia, remoteMedia);
var videoStream = new FM.IceLink.VideoStream(localMedia, remoteMedia);
LocalCameraMedia localMedia = new LocalCameraMedia();
RemoteMedia remoteMedia = new RemoteMedia();

fm.icelink.AudioStream audioStream = new fm.icelink.AudioStream(localMedia, remoteMedia);
fm.icelink.VideoStream videoStream = new fm.icelink.VideoStream(localMedia, remoteMedia);
LocalCameraMedia* localMedia = [LocalCameraMedia new];
RemoteMedia* remoteMedia = [RemoteMedia new];

FMIceLinkAudioStream* audioStream = [FMIceLinkAudioStream audioStreamWithLocalMedia:localMedia remoteMedia:remoteMedia];
FMIceLinkVideoStream* videoStream = [FMIceLinkVideoStream videoStreamWithLocalMedia:localMedia remoteMedia:remoteMedia];
var localMedia = LocalCameraMedia()
var remoteMedia = RemoteMedia()

var audioStream = FMIceLinkAudioStream(localMedia: localMedia, remoteMedia: remoteMedia)
var videoStream = FMIceLinkVideoStream(localMedia: localMedia, remoteMedia: remoteMedia)
var localMedia = new fm.icelink.LocalMedia(true, true);
var remoteMedia = new fm.icelink.RemoteMedia();

var audioStream = new fm.icelink.AudioStream(localMedia, remoteMedia);
var videoStream = new fm.icelink.VideoStream(localMedia, remoteMedia);


As mentioned above, this is the most common configuration for streams. However, in some cases, you may only wish to send or receive data. The next section covers one-way, or unidirectional streams.

Creating a One-Way Stream

A one-way stream is a stream in which either data is only sent or data is only received. This type of stream has only a single end-point. Logically then, to create a one-way stream, you simply omit either local or remote media when you create your AudioStream and VideoStream instances. To create a receive-only stream, specify null instead of a LocalCameraMedia instance, as shown below:

var remoteMedia = new RemoteMedia();

var videoStream = new FM.IceLink.VideoStream(null, remoteMedia);
RemoteMedia remoteMedia = new RemoteMedia();

fm.icelink.VideoStream videoStream = new fm.icelink.VideoStream(null, remoteMedia);
RemoteMedia* remoteMedia = [RemoteMedia new];

FMIceLinkVideoStream* videoStream = [FMIceLinkVideoStream videoStreamWithLocalMedia:nil remoteMedia:remoteMedia];
var remoteMedia = RemoteMedia()

var videoStream = FMIceLinkVideoStream(localMedia: nil, remoteMedia: remoteMedia)
var remoteMedia = new fm.icelink.RemoteMedia();

var videoStream = new fm.icelink.VideoStream(null, remoteMedia);


The opposite of a receive-only stream is a send-only stream. In this case, you provide your LocalCameraMedia instance, but you specify null instead of a RemoteMedia instance:

var localMedia = new LocalMedia();

var videoStream = new FM.IceLink.VideoStream(localMedia, null);
LocalMedia localMedia = new LocalMedia();

fm.icelink.VideoStream videoStream = new fm.icelink.VideoStream(localMedia, null);
LocalMedia* localMedia = [LocalMedia new];

FMIceLinkVideoStream* videoStream = [FMIceLinkVideoStream videoStreamWithLocalMedia:localMedia remoteMedia:nil];
var localMedia = LocalMedia()

var videoStream = FMIceLinkVideoStream(localMedia: localMedia, remoteMedia: nil)
var localMedia = new fm.icelink.LocalMedia(true, true);

var videoStream = new fm.icelink.VideoStream(localMedia, null);


One-way streams are great if you know that you will never need to send or receive data. However, there are some cases where you want to stop sending or receiving data for a set period of time. To do this, read the next section on switching the direction of a stream.

Creating a Data Stream

Audio and video streams are perfect for media data, but what if you want to add chat messages or file transfers to your video conference? The IceLink SDK enables this and allows you to send any arbitrary text or byte data over what is known as a data stream.

Configuring a data stream has one additional step beyond configuring audio and video streams. Each data stream has one or more data channels, which can be configured to send a specific type of data. If you think of a data stream as a highway, think of each data channel as a separate lane. You might have one data channel for chat messages, another for file transfers and even another for system messages, such as letting users know when participants are away from their keyboard. Data channels support transfer of short message binary or text data. This could be useful for transferring any sort of data outside of the typical audio/video use case typically associated with RTC.

File Transfer

Our current implementation of DataChannels is optimized for small chunks of binary or text messages rather than for file transmission. While sending large files is possible, the rate will be restricted statically as we do not yet have full dynamic congestion control in place. At the moment, we are working on the Partial Reliability extension of SCTP (https://tools.ietf.org/html/rfc3758). Fast data transmission rates for large portions of data (such as for file transfer) will only be recommended after full implementation of congestion control.

Working with DataStreams

Before you can create a data stream, you need to create data channels for it. To create a data channel, create an instance of FM.IceLink.DataChannel. Provide a label for the channel as the first parameter, which can be any string. Next, add an OnReceive event handler to the data channel. This event handler fires whenever a message is received on this data channel. If you expect the message to be plain text, you can access it using the DataString property of the event handler arguments. If you expect the message to be raw binary data, you can access it using the DataBytes property of the event handler arguments. IceLink does not know anything about the nature of data you are sending, so it is up to you to make sure you use the correct property.

Once you have a DataChannel instance, you can create a data stream. To create a DataStream instance, call its constructor and specify the data channel as the first parameter. This will make the data stream and all of its data channels available to all participants in a session.

The following code block shows how to create a data stream:

var dataStream = new FM.IceLink.DataStream(dataChannel);
fm.icelink.DataChannel dataChannel = new fm.icelink.DataChannel("chat-channel");

dataChannel.setOnReceive((fm.icelink.DataChannelReceiveArgs e) -> {
    // for text data
    String data = e.getDataString();
    
    // for binary data
    byte[] data = e.getDataBytes().getData();
});
fm.icelink.DataStream dataStream = new fm.icelink.DataStream(channel);

FMIceLinkDataChannel* dataChannel = [FMIceLinkDataChannel dataChannelWithLabel:@"chat-channel"];

[dataChannel setOnReceiveBlock: ^(FMIceLinkDataChannelReceiveArgs* e) {
    // for text data
    NSString* data = [e dataString];
    
    // for binary data
    NSData* data = [[e dataBytes] data];
}];
FMIceLinkDataStream* dataStream = [FMIceLinkDataStream dataStreamWithChannel:dataChannel];
var dataChannel = FMIceLinkDataChannel(label: "chat-channel")

dataChannel.setOnReceiveBlock { (e:FMIceLinkDataChannelReceiveArgs) in
    // for text data
    NSString data = e.dataString()
    
    // for binary data
    NSData data = e.dataBytes().data()
}
var dataChannel = FMIceLinkDataChannel(label: "chat-channel")

dataChannel.setOnReceiveBlock { (e:FMIceLinkDataChannelReceiveArgs) in
    // for text data
    NSString data = e.dataString()

    // for binary data
    NSData data = e.dataBytes().data()
}

var dataStream = FMIceLinkDataStream(channel: dataChannel)
var dataChannel = new fm.icelink.DataChannel("chat-channel");

dataChannel.setOnReceive(function(e) {
    // for text data
    var data = e.getDataString();

    // for binary data
    var data = e.getDataBytes().getData();
});

var dataStream = new fm.icelink.DataStream(channel);

Once a data stream has been established, you can send messages on any of its associated data channels using the SendDataBytes and SendDataString methods of a DataChannel instance. This means that you will need to keep a reference to any data channels that you create. Note that like audio and video streams, data streams are between two end-points. This means that for each participant in a conference, you will have one data stream and one set of data channels. If you want to send data to a specific user, you must send data on the data channel associated with the user. If you want to send data to all users, you must send data on each user's associated data channel.

This can get tricky when you have multiple types of data channels. To help with this, IceLink provides the FM.IceLink.DataChannelCollection class. You will usually create one instance of this for each type of data channel that your session supports. When you want to send data to all participants, you iterate over the collection, and send to all active data channels. An example of this is shown below:


var dataChannelCollection = new FM.IceLink.DataChannelCollection();

foreach (FM.IceLink.DataChannel dataChannel in dataChannelCollection.Values)
{
    // for text data
    dataChannel.SendDataString("my-data");

    // for binary data
    dataChannel.SendDataBytes(new byte[0]);
}
fm.icelink.DataChannelCollection dataChannelCollection = new fm.icelink.DataChannelCollection();

for (fm.icelink.DataChannel dataChannel : dataChannelCollection.getValues()) {
    // for text data
    dataChannel.sendDataString("my-data");

    // for binary data
    dataChannel.sendDataBytes(new byte[0]);
}
FMIceLinkDataChannelCollection* dataChannelCollection = [FMIceLinkDataChannelCollection dataChannelCollection];

for (FMIceLinkDataChannel* dataChannel in [dataChannelCollection values]) {
    // for text data
    [dataChannel sendDataStringWithDataString: @"my-data"];

    // for binary data
    [dataChannel sendDataBytesWithDataBytes: [[NSData alloc] init]];
}
var dataChannelCollection = FMIceLinkDataChannelCollection()

for dataChannel in dataChannelCollection.values() {
    // for text data
    dataChannel.sendDataString(dataString: "my-data")

    // for binary data
    dataChannel.sendDataBytes(dataBytes: NSData())
}
var dataChannelCollection = new fm.icelink.DataChannelCollection();

dataChannelCollection.getValues().forEach(function(dataChannel) {
    // for text data
    dataChannel.sendDataString("my-data");

    // for binary data
    dataChannel.sendDataBytes([]);
});


Each data channel also has an OnStateChange event, which is useful for managing the contents of your DataChannelCollection instances. You can add the data channel to the collection when you it has opened and remove the data channel when it has closed. This ensures that when you are iterating over the collection, you never attempt to send data to a closed data channel. This is demonstrated below:

dataChannel.OnStateChange += (FM.IceLink.DataChannel dataChannel) =>
{
    if (dataChannel.State == FM.IceLInk.DataChannelState.Connected)
    {
        dataChannelCollection.Add(dataChannel);
    }
    else if (dataChannel.State == FM.IceLink.DataChannelState.Closed || dataChannel.State == FM.IceLink.DataChannelState.Failed)
    {
        dataChannelCollection.Remove(dataChannel);
    }
};
dataChannel.addOnStateChange((fm.icelink.DataChannel dataChannel) -> {
    if (dataChannel.getState() == fm.icelink.DataChannelState.Connected) {
        dataChannelCollection.add(dataChannel);
    } else if (dataChannel.getState() == fm.icelink.DataChannelState.Closed || dataChannel.getState() == fm.icelink.DataChannelState.Failed) {
        dataChannelCollection.remove(dataChannel);
    }
});
[dataChannel addOnStateChangeWithBlock: ^(FMIceLinkDataChannel* dataChannel) {
    if (dataChannel.state == FMIceLinkDataChannelStateConnected) {
        [dataChannelCollection addWithValue: dataChannel];
    } else if (dataChannel.state == FMIceLinkDataChannelStateClosed || dataChannel.state == FMICeLinkDataChannelStateFailed) {
        [dataChannelCollection removeWithValue: dataChannel];
    }
}];
dataChannel.addOnStateChange { (channel:FMIceLinkDataChannel) in
    if channel.state == FMIceLinkDataChannelStateConnected {
        dataChannelCollection.add(value: dataChannel)
    } else if channel.state == FMIceLinkDataChannelStateClosed || channel.state == FMIceLinkDataChannelStateFailed {
        dataChannelCollection.remove(value: dataChannnel)
    }
}
dataChannel.addOnStateChange(function(dataChannel) {
    if (dataChannel.getState() == fm.icelink.DataChannelState.Connected) {
        dataChannelCollection.add(dataChannel);
    } else if (dataChannel.getState() == fm.icelnk.DataChannelState.Closed || dataChannel.getState() == fm.icelink.DataChanenlState.Failed) {
        dataChannelCollection.remove(dataChannel);
    }
});

Wrapping Up

You've learned about each individual component in the IceLink stack. The final step is to put it all together to establish a peer-to-peer video conference. The next section describes how to do this by Opening a Connection.