Xamarin iOS

API Reference

The Xamarin API ref documentation is available here.

Starting a Xamarin iOS Project

Xamarin iOS uses AOT (Ahead of time) compilation, as opposed to other C# targets, which use JIT (Just in time) compilation. Because of this, you must ensure that Monotuch links to the native libraries so that they are available at compile-time. To do this, add the following MSBuild directive to your Xamarin iOS .csproj file:

<MtouchExtraArgs>-gcc_flags "-L${ProjectDir}/libs/native -lvpxfm-iOS -lopusfm-iOS -lyuvfm-iOS -force_load ${ProjectDir}/libs/native/libvpxfm-iOS.a -force_load ${ProjectDir}/libs/native/libopusfm-iOS.a -force_load ${ProjectDir}/libs/native/libyuvfm-iOS.a"</MtouchExtraArgs>

If you don't want to edit the .csproj file directly, you can instead specify this through the IDE. Open the project property window and look for the "Monotouch Arguments" property. Copy the contents of the MTouchExtraArgs element here.

If you do not specify this, then Xamarin iOS won't compile for physical devices and there will be run-time exceptions when the application is run on the iOS simulator.

As with other platforms, you will need to include architecture-specific native libraries for Xamarin iOS. Include the libraries that can be found in the Xamarin/Libraries/iOS/native folder. The following rules apply:

  • If you are using FM.LiveSwitch.Opus.dll, include libopusfm-iOS.a and libopus-iOS.a.
  • If you are using FM.LiveSwitch.Vpx.dll, include libvpxfm-iOS.a and libvpx-iOS.a.
  • If you are using FM.LiveSwitch.Yuv.dll, include libyuvfm.iOS.a and libyuv-iOS.a.

If you want to be safe, you can include the entire lib folder in your project. This will ensure that you will always have the correct native libraries.

Mono Dynamic Registrar Flag

If you are using Mono 5.10.x or higher there is a know issue with Xamarin iOS where the dynamic linker is removed at runtime. The native libraries will fail to load without the dynamic linker. A work around is to add "--optimize=-remove-dynamic-registrar" flag to MTouchExtraArgs in the .csproj file . Adding this flag will prevent Mono from removing the dynamic linker.

After you add this in your .csproj file, your .cjproj file should have the following:

<MtouchExtraArgs>--optimize=-remove-dynamic-registrar -gcc_flags "-L${ProjectDir}/libs/native -lvpxfm-iOS -lopusfm-iOS -lyuvfm-iOS -force_load ${ProjectDir}/libs/native/libvpxfm-iOS.a -force_load ${ProjectDir}/libs/native/libopusfm-iOS.a -force_load ${ProjectDir}/libs/native/libyuvfm-iOS.a"</MtouchExtraArgs>

Setting Up Client Logging

The LiveSwitch Client SDK has an internal logging API that its components use to provide you with information about what's happening. You will want to hook into this API at some point, because you will want to know when an error occurs. This guide describes how the logging API works and how you can integrate your application with it.

Understanding the Logging Model

The logging API outputs messages with one of five possible severity levels. The severity level indicates the importance of the message. The levels are, from most severe to least severe:

  • FATAL: The message indicates a fatal application error; the application will need to shut down and no recovery actions are possible.
  • ERROR: The message indicates a normal application error (ie: timeouts, bad user input); this is a normal error state and usually it is possible to proceed.
  • WARN: The message indicates that something abnormal occurred but that the application was not negatively affected; this is used for unexpected values, deprecations or abnormal application state.
  • INFO: This is a trace message; it describes what is going on in the SDK.
  • DEBUG: This is a diagnostic message; it is similar to the INFO message but is used for outputting detailed parameters or for "spammy" messages.

You can output a log message by invoking one of several methods on the static FM.LiveSwitch.Log class. The methods are named after the severity level of the logs they output. They are DebugInfoWarnErrorFatal. The following snippet shows how to output a DEBUG and an ERROR log message.

You can also set the level of logs you want displayed. By default, only INFO and more severe messages are displayed. In production, you may want to display no message, and in development, you may want to display all diagnostic messages. This is accomplished by setting the LogLevel property of the FM.LiveSwitch.Log class. Note that there is one additional level here, None. You can set this if you want to suppress all log messages.

Log Levels
FM.LiveSwitch.Log.Debug("This is a DEBUG message.");
FM.LiveSwitch.Log.Fatal("This is a FATAL message.");
Setting Log Levels
// for development
FM.LiveSwitch.Log.LogLevel = FM.LiveSwitch.LogLevel.Debug;


// for production
FM.LiveSwitch.Log.LogLevel = FM.LiveSwitch.LogLevel.None;
// or
FM.LiveSwitch.Log.LogLevel = FM.LiveSwitch.LogLevel.Error;

Registering a Provider

By default, nothing happens when a log message is generated. You must first specify what should happen when a log message is output. This is done by registering an FM.LiveSwitch.LogProvider instance with the static FM.LiveSwitch.Log class. Each provider has a specific method of outputting a log message. A common provider is one that outputs messages to the log console but you can also create providers that output messages to files or to a cloud logging provider.

Each provider has an associated severity level. These severity levels are used in combination with the global severity level to determine whether or not a log message should be output. This allows you to specify that some log providers with an expensive logging mechanism (ie: a cloud log provider) should only log error messages, while log providers with a fast logging mechanism can output everything, including debug messages.

Adding a provider is accomplished by invoking the RegisterProvider method of the static FM.LiveSwitch.Log class. This example uses the default LiveSwitch log providers, all of which output to the console. Note how the provider's log level is specified when the provider is instantiated.

FM.LiveSwitch.Log.Level = FM.LiveSwitch.LogLevel.Debug;
FM.LiveSwitch.Log.RegisterProvider(new FM.LiveSwitch.ConsoleLogProvider(FM.LiveSwitch.LogLevel.Debug));

Removing a Provider

Generally, you will not want to remove a provider. Log providers are usually set up on application start up and there are not many use cases for changing them during execution. However, LiveSwitch does provide this functionality. You will need to retain a reference to the specific provider you wish to remove. Pass this reference into the RemoveProvider method of the static FM.LiveSwitch.Log instance.

FM.LiveSwitch.Log.RemoveProvider(consoleLogProvider);

Implementing Your Own Log Provider

LiveSwitch provides some basic FM.LiveSwitch.LogProvider implementations but you may want to create some application-specific implementations. To do so, extend the LogProvider class and implement the DoLog method. The method has five parameters:

  • timestamp: The date and time when the message was logged.
  • level: The severity level of the message.
  • message: The actual text of the log message.
  • tag: A value based on which component generated the log message.
  • ex: If the log message is associated with an exception, then this will contain the exception instance; otherwise it will be null.

If you do not need to perform any custom formatting on the message, it's recommended to use the inherited GenerateLogLine method. This returns a string in the same format as the other LiveSwitch log providers. You can then output the message in whatever way you want. The following code shows a re-implementation of the console log provider.

Note: The LogProvider base class defaults to the INFO log level. If you want to support a different logging level in your custom log provider be sure to pass the logging level through in the constructor.

public class ConsoleLogProvider : FM.LiveSwitch.LogProvider
{
    protected override void DoLog(DateTime timestamp, FM.LiveSwitch.LogLevel level, string tag, string message, Exception ex)
    {
        Console.WriteLine(GenerateLogLine(timestamp, level, tag, message, ex);
    }
}

You now have all the knowledge that you need to incorporate the LiveSwitch logging API into your client application. If you prefer to use another logging framework, that's perfectly fine. In this case, you can write a custom log provider to send LiveSwitch log messages to the logging framework and let the framework handle the output of messages.

Registering a Client

In order to participate in a video conference, you need some way for the participants to communicate with each other. In LiveSwitch, the LiveSwitch Gateway fulfills this role. The gateway connects everyone who wants to participate in a media session. It controls which individuals can communicate with each other and restricts unauthorized access. To start a session, you must first register with the LiveSwitch gateway. This section describes the registration process and outlines some of the core concepts associated with session management.

Authorization Tokens

When you submit a request to register with the LiveSwitch Gateway, you will provide an authorization token. A token is an key that is encoded with the details of your authorization request. Each token is composed of several parts, which are described here in detail:

  • Application ID: A unique identifier specific to an application. The LiveSwitch server can accommodate multiple simultaneous applications, and this parameter is used to distinguish between them. If you only have a single application, you can hard-code this value.
  • User ID: A unique identifier for the user who wishes to register with the server, such as a username.
  • Device ID: Generally, this is a random GUID that uniquely identifies which device the user is on. It is used to distinguish between multiple sessions, if a user is connected on multiple devices.
  • Client ID: A unique value generated when instantiating a Client instance. This is generally only used if there are multiple clients running in a single application.
  • Roles: In some applications, different users are given different sets of permissions. For example, a distance learning application might have a "student" role and a "teacher" role. You can omit this if your application does not require it. If you do use roles, then the client must specify their roles in the .ctor, and the token roles are used to ensure that a given client has permission to assume that set of roles. If a client attempts to assume a role that is not included by their registration token, then registration will fail.
  • Channels: A channel is a unique identifier that describes an audio or video conference. Generally, for each video conference that you participate in, you will have a unique channel associated with it.
  • Secret Key: This is a secret that is shared with the server, and is used to validate registration requests. You can begin by hard-coding this but you should not hard-code this in production. It can be any arbitrary value.
  • Next, you will create one of these tokens and use it to register with the LiveSwitch Gateway.

Binding Hierarchy

For separating signalling concerns for different applications, channels, users, etc., LiveSwitch uses the combination of the above IDs hierarchically: Application ID / Channel ID / User ID / Device ID / Client ID. This binding hierarchy uniquely binds the LiveSwitch signalling client, for a given device and a given user, to a channel for a given application. As you can see this allows maximum flexibility. You can have multiple channels per application, within a channel you can have multiple users, or alternatively you could have the same user but this user can be using different devices. Whatever the use case, the binding hierarchy allows you enough flexibility to meet your requirements for uniquely identifying your signalling concerns.

Registering with the Gateway

Before you can start a media session, you must create a client and register with the LiveSwitch Gateway. As mentioned above, the first part of registration requires you to create an authorization token. The server may accept or reject the registration request based on the value of the token. The next snippet of code looks at how to create a token for a simple registration request. The request has no roles, and only asks to join a single channel. The secret is also hard-coded for simplicity.

You should never generate the token client side in production. We're doing so for demo purposes ONLY. Refer to the section on Creating an Auth Server for more information.

var applicationId = "my-app";
var userId = "my-name";
var deviceId = "00000000-0000-0000-0000-000000000000";
var channelId = "11111111-1111-1111-1111-111111111111";

var client = new FM.LiveSwitch.Client("http://localhost:8080/sync", applicationId, userId, deviceId, null, new[] { "role1", "role2" });

string token = FM.LiveSwitch.Token.GenerateClientRegisterToken(
    applicationId,
    client.UserId,
    client.DeviceId,
    client.Id,
    client.Roles,
    new[] { new FM.LiveSwitch.ChannelClaim(channelId) },
    "--replaceThisWithYourOwnSharedSecret--"
);

Once you have created a token, the next step is to make a registration request. This is done by invoking the Register method of the FM.LiveSwitch.Client instance that you created. This method returns a promise, which is resolved if registration is successful and is rejected if it is not. You should always validate the result of this promise before attempting to proceed with any further operations. If the registration is successful, then the promise will return a set of channels that it has registered the client to. These will match the channels that you specified when you created your authorization token.

At this point, you would normally add a variety of event handlers to the FM.LiveSwitch.Channel instances. This is covered in the following sections, Opening an MCU Connection, Opening an SFU Connection and Opening a Peer-to-Peer Connection. For now, focus on completing the registration, as shown in the sample below.

client.Register(token).Then((FM.LiveSwitch.Channel[] channels) =>
{
    Console.WriteLine("connected to channel: " + channels[0].Id);
}).Fail((Exception ex) =>
{
    Console.WriteLine("registration failed");
});

Note that this section assumes that you know which channels you want to join when you want to register. This is not always the case. The next section, Joining a Channel, describes how to join and leave channels after you have already registered with the LiveSwitch Gateway.

Unregistering

It is a best practice to unregister from the LiveSwitch Gateway before closing your application. Although the gateway will automatically unregister clients that are inactive for long periods of time, this will ensure that resources are cleaned up in a timely fashion and that other clients are properly notified that a client has disconnected. To unregister a client, invoke the Unregister method of your Client instance. Like the Register method, this also returns a promise, which you can inspect to see if unregistration is successful. Generally, this is not necessary, as you only perform this if you are exiting the application anyways.

client.Unregister().Then((object result) =>
{
    Console.WriteLine("unregistration succeeded");
}).Fail((Exception ex) =>
{
    Console.WriteLine("unregistration failed");
});

You are now able to connect your client-side applications with your LiveSwitch Gateway instance by generating a token and registering with the gateway. This is the first step towards establishing a video conference. Remember that although we have only covered one way of joining channels - at registration - that you can also join channels at any point in time afterwards. This is covered in the next section, Joining a Channel.

Joining a Channel

In the previous section on Registering a Client, you learned how to create a token and register a client with the LiveSwitch Gateway. The examples in that section assumed that you knew, in advance, the channels that you wanted your client to join. This is not a realistic assumption - a client may wish to join and leave channels after it has already registered or it may wish to leave a channel that is has previously joined when registering. This section goes into further detail about how to manage a client's channels.

Adding a New Channel

Joining a channel requires generating a token, the same way you generated a token when registering with the LiveSwitch Gateway. The syntax for doing so is similar but slightly different. First, instead of specifying an array of FM.LiveSwitch.ChannelClaim instances; you can only specify a single channel to join. Secondly, you do not specify roles when generating a token to join a channel, as these must be provided when registering. Finally, instead of GenerateClientRegisterToken, you use the GenerateClientJoinToken method of the FM.LiveSwitch.Token class. The code below shows how you generate a token token to join a new channel.

As with creating registration tokens, creating tokens to join a channel should never be done client side. Refer to the section on Creating an Auth Server for more information.

var applicationId = "my-app";
var userId = "my-name";
var deviceId = "00000000-0000-0000-0000-000000000000";
var channelId = "99999999-9999-9999-9999-999999999999";

var client = new FM.LiveSwitch.Client(...);

string token = FM.LiveSwitch.Token.GenerateClientJoinToken(
    applicationId,
    client.UserId,
    client.DeviceId,
    client.Id,
    new FM.LiveSwitch.ChannelClaim(channelId),
    "--replaceThisWithYourOwnSharedSecret--"
);

Once you have created your token, the next step is to validate this token with the LiveSwitch Gateway. To do this, invoke the Join method of your FM.LiveSwitch.Client instance using the token and a channel id. The channel id that you specify must match the id of the ChannelClaim instance that you used to create the token. The Join method will return a promise, which, as usual, you should inspect to ensure that you have joined the channel successfully.

Following this, you have successfully joined the channel and will now receive messages on it. The next section covers leaving a channel.

client.Join(channelId, token).Then((FM.LiveSwitch.Channel channel) =>
{
    Console.WriteLine("successfully joined channel");
}).Fail((Exception ex) =>
{
    Console.WriteLine("failed to join channel");
});

Leaving a Channel

A client may wish to leave a channel for many reasons, such as when a video conference ends. To leave a channel, you invoke the Leave method of your FM.LiveSwitch.Client instance, with the channelId of the channel you wish to leave. The Leave method returns a promise, which, as demonstrated below, you should inspect to see the if you have successfully left the channel.

client.Leave(channelId).Then((FM.LiveSwitch.Channel channel) =>
{
    Console.WriteLine("left the channel");
}).Fail((Exception ex) =>
{
    Console.WriteLine("failed to leave the channel");
});

Following this, you have now successfully left the channel. Note that if you do leave a channel, your previous authorization token for this channel is invalid. You must generate a new token to join the same channel.

You can now join and leave channels at any point in your application. Channels are flexible components of LiveSwitch, and you can use them for many things - as individual channels, as notification subscriptions, or whatever else your application needs. This guide hasn't covered every possible use of a channel, so feel free to play around with the API and see what you can do with them.

Handling Local Media

To send media in a video conference, you need to have some way to produce local audio/video. The audio and/or video produced and sent by the current user is referred to as local media, because it is local to the user doing the sending. This section focuses on how to capture local media for sending to remote participants.

The local media abstraction in the client SDK covers the most common use case for producing audio and video data. It wraps a single audio track and a single video track. It is possible to bypass the media abstraction and work with tracks directly, which are composed of a sequence of media processing elements. It is also possible to bypass the track abstraction and work directly with media sources and pipes, connecting and disconnecting them at runtime to support advanced use cases. For the purposes of this guide, we will focus on the media abstraction, since it addresses the most common use case.

Defining the Local Media Object

The first step to capturing local media is to define how this capture will be performed. This is done by extending the FM.LiveSwitch.RtcLocalMedia<TView> class. In this context, the generic type TView represents the type of object that will be used for displaying the video preview. For a concrete example here, which focuses on camera capture, we will use the concrete FM.LiveSwitch.Cocoa.OpenGLView type for displaying a local preview.

You can have multiple types of local media implementations per application, though generally, you will only have one. This is because each local media implementation is usually associated with a specific set of inputs and the most common set of inputs is a camera and a microphone. To that end, this guide focuses on capturing the user's camera and microphone using an implementation of RtcLocalMedia named LocalMedia. The LocalMedia class will be prefixed by an App namespace, so that it's clear the the class resides in the application layer and is not part of the LiveSwitch SDK.

To extend RtcLocalMedia, you must implement several factory methods. These methods create the components that capture and process your media data. If you are not using the specific component returned by the method, you can return null. This guide will indicate which components are required for specific operations, so you can make this determination on your own. Note that you will never invoke these methods from your application code - LiveSwitch itself will invoke them when necessary. Your only responsibility here is to provide an implementation for these methods.

Note that the _Preview property here is necessary for displaying a preview of the camera. It is instantiated in the constructor here so that we have access to it elsewhere. This is covered in more depth in the following section Capturing Local Video.

public class LocalMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
	private FM.LiveSwitch.Cocoa.AVCapturePreview _Preview;

    public LocalMedia(bool disableAudio, bool disableVideo, AecContext aecContext)
        : base(disableAudio, disableVideo, aecContext)
    {
		_Preview = new FM.LiveSwitch.Cocoa.AVCapturePreview();
        Initialize();
    }
}

Capturing Local Audio

To enable audio for your App.LocalMedia class, you must implement two methods. The first method, CreateAudioSource, returns an audio source that will be used for recording audio. For the purposes of this example, the audio source will be the user's microphone. LiveSwitch provides a library for your platform that allows you to capture audio from the user's microphone. This is included in the LiveSwitch distribution:

  • For Xamarin iOS: include FM.LiveSwitch.Cocoa.dll.
public class LocalMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.AudioSource CreateAudioSource(FM.LiveSwitch.AudioConfig config)
    {
        return new FM.LiveSwitch.Cocoa.AudioUnitSource(config);
    }
}

The second method to implement is the CreateOpusEncoder method. Technically, this isn't required, but if you do not implement it, your application will be forced to fall back to the lower quality PCMA/PCMU audio codecs. The code samples below demonstrates how to enable the Opus encoder. Once you complete this your application will be able to capture the user's audio and send it to other participants. Similar to above, LiveSwitch provides platform-specific libraries under the FM.LiveSwitch.Opus namespace. Add the corresponding library to your project:

  • For Xamarin iOS: include FM.LiveSwitch.Opus.dll.
public class LocalMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.AudioEncoder CreateOpusEncoder(FM.LiveSwitch.AudioConfig config)
    {
        return new FM.LiveSwitch.Opus.Encoder(config);
    }
}

Capturing Local Video

Capturing video for the App.LocalMedia class works the same as it does for audio but there are a few more steps. In addition to creating a video source and a video encoder, you also need to provide factory methods to create a video preview and to create some image manipulation tools. To start, however, provide an implementation for the CreateVideoSource method. LiveSwitch provides implementations for a video source that captures video data from the user's camera. For Xamarin you'll have to include FM.LiveSwitch.Cocoa.dll.

The code sample below shows how to create your source. For iOS a native object for displaying a preview of the user's camera is required. You will need to access this, and pass it into the video source that you create. If you recall we initialized this earlier in the local media constructor above.

// In the constructor above we instantiated the _Preview property.
// The _Preview is required to create your AVCaptureSource.
public class LocalMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.VideoSource CreateVideoSource()
    {
        return new FM.LiveSwitch.Cocoa.AVCaptureSource(_Preview, new FM.LiveSwitch.VideoConfig(640, 480, 30));
    }
}

If you remember from the above section, we created an object to use the platform's native camera preview. Here we provide that object as a view that displays the data from the preview instance that you created previously. We must also return null from the CreateViewSink function as shown here, so that the view we have created is used directly.

public class LocalMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
	// In the constructor above we instantiated the _Preview property.
	public UIKit.UIView GetView()
	{
		return _preview;
	}


	// We return null so that our _Preview is used instead of a view sink.
    public override FM.LiveSwitch.ViewSink<FM.LiveSwitch.Cocoa.OpenGLView> CreateViewSink
    {
		return null;
    }
}

The next step is to specify which video codecs to use. This is done by providing implementations for the CreateVp8EncoderCreateVp9Encoder and CreateH264Encoder classes. These implementations can be found in the FM.LiveSwitch.Vpx and FM.LiveSwitch.OpenH264 libraries. If you want to support a codec, return an appropriate encoder from the factory method associated with a codec. You can instead disable a codec, by returning null from the its factory method. The code below demonstrates how to enable these codecs.

  • For Xamarin iOS: include FM.LiveSwitch.Vpx.dll.
  • Note that for Mac platforms, there is no OpenH264 library, as these platforms natively support the H264 codec.
public class LocalMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.VideoEncoder CreateVp8Encoder()
    {
        return new FM.LiveSwitch.Vp8.Encoder();
    }
    public override FM.LiveSwitch.VideoEncoder CreateVp9Encoder()
    {
        return new FM.LiveSwitch.Vp9.Encoder();
    }
    public override FM.LiveSwitch.VideoEncoder CreateH264Encoder()
    {
        return new FM.LiveSwitch.H264.Encoder();
    }
}

Finally, you must provide a minor image formatting utility by implementing the CreateImageConverter method. This creates a tool that converts between various color spaces, which are different ways of representing colors. This is needed because webcams do not capture data in the i420 color space, which is required by the LiveSwitch video encoders. You should return an instance of FM.LiveSwitch.Yuv.ImageConverter for this method. The code below demonstrates how to create this object. At this point, your App.LocalMedia class can capture and send both audio and video. Once you have completed this the next step is to actually kick off the process of capturing data.

As the class name indicates, this classes can be found in the FM.LiveSwitch.Yuv library. The library itself is a small wrapper around libyuv. LiveSwitch provides compiled versions of this library for your platform:

  • For Xamarin iOS: include FM.LiveSwitch.Yuv.dll.
public class LocalMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.VideoPipe CreateImageConverter(FM.LiveSwitch.VideoFormat outputFormat)
    {
        return new FM.LiveSwitch.Yuv.ImageConverter(outputFormat);
    }
}

Controlling Media Capture

To start capturing media, invoke the Start method of your App.LocalMedia implementation. This method is inherited from the FM.LiveSwitch.RtcLocalMedia<T> base class; you don't have to implement it yourself. The Start method returns a promise, which will resolve when the instance begins to capture data from the user's camera and microphone. If media can't be captured, then the promise will be rejected. You should always specify a reject action, so that you can be notified if an error occurs. Starting media capture is demonstrated below.

localMedia.Start().Then((FM.LiveSwitch.LocalMedia lm) =>
{
    Console.WriteLine("media capture started");
}).Fail((Exception ex) =>
{
    Console.WriteLine(ex.Message);
});

Stopping capture works the same way. Invoke the Stop method of your App.LocalMedia class to stop capturing camera and microphone data. Again, this will return a promise, which you can use to determine when the media capture has stopped.

localMedia.Stop().Then((FM.LiveSwitch.LocalMedia lm) =>
{
    Console.WriteLine("media capture stopped");
}).Fail((Exception ex) =>
{
    Console.WriteLine(ex.Message);
});

You can start and stop your local media instance as many times as necessary. When you are completely finished with the local media instance, you should destroy it by invoking its Destroy method. This ensures that any input devices are released back to the system.

localMedia.Destroy();

Displaying a Local Preview

Now that you know how to capture data from a user's camera and microphone, you want to be able to show the user a preview of the video that they are sending out. The easiest way to do this is to let LiveSwitch handle it. You can do this by assigning the user's local media to an instance of FM.LiveSwitch.LayoutManager, which is a class to manage the local and remote video feeds for a video conference.

The specific LayoutManager instance that you should use will vary based on your platform but all of the implementations work in largely the same way. Before a video session starts, you create an instance of your layout manager class and pass it a container parameter. The container is used as a canvas for the layout manager and controls the dimensions of displayed video. For example, if you assign the layout manager a 400x400 container and there are four people in a video session, the layout manager will draw each person's video in a 200x200 block.

The code samples below show how to initialize a LayoutManager instance for each platform. Note that the API is identical except for the types of objects used.

var layoutManager = new FM.LiveSwitch.Cocoa.LayoutManager((UIKit.UIView)container);

Once you have created a LayoutManager instance, you can assign the local view from your App.LocalMedia instance to it. You first retrieve the view by accessing the View property of the App.LocalMedia instance. This returns an object of an appropriate type for your platform. You can now assign this view to the layout manager by invoking the SetLocalView method, demonstrated below.

layoutManager.SetLocalView(localMedia.View);

When you are done with a media session, you will also want to remove this view from the layout manager. This is done by invoking the UnsetLocalView method of the layout manager.

layoutManager.UnsetLocalView();

Handling Remote Media

To receive media in a video conference, you need to have some way to consume the remote audio/video. The audio and/or video received by the current user is known as remote media, because it comes from a remote participant. This section focuses on how to play back the media received from remote participants.

The remote media abstraction in the client SDK covers the most common use cases for consuming audio and video data. It wraps a single audio track and a single video track. It is possible to bypass the media abstraction and work with tracks directly, which are composed of a sequence of media processing elements. It is also possible to bypass the track abstraction and work directly with media pipes and sinks, connecting and disconnecting them at runtime to support advanced use cases. For the purposes of this guide, we will focus on the media abstraction, since it addresses the most common use case.

Defining the Remote Media Object

Similar to how a user's local media require an implementation that inherits from FM.LiveSwitch.RtcRemoteMedia<T>, other participants' remote media require an implementation that inherits from FM.LiveSwitch.RtcRemoteMedia<T>. The generic type T represents the type of object that will be used for displaying the video feeds of these remote participants. As with local media you do not have to specify a generic type.

When working with local media, you created a class named App.LocalMedia. For remote media, continue this convention by creating a class named App.RemoteMedia. You will find that the implementation for both objects is similar.

Initializing the Remote Media Object

To begin implementing App.RemoteMedia, define a constructor. Your constructor must call one of the parent constructors in the RtcRemoteMedia<T> class and must invoke the Initialize method. There are two parent constructors that you can choose from. The first has two parameters, disableAudio and disableVideo, which allow you to disable either the audio or video data from a remote feed. It is recommended to leave both audio and video enabled for maximum flexibility and interoperability with the remote peer. The second constructor, takes an instance of FM.LiveSwitch.AecContext, which, again, is short for Acoustic Echo Cancellation. The implementation of the second constructor is out of scope for this guide, so focus on the first, as shown in the following code samples:

public class RemoteMedia : FM.LiveSwitch.RtcRemoteMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public RemoteMedia(bool disableAudio, bool disableVideo, AecContext aecContext)
        : base(disableAudio, disableVideo, aecContext)
    {
        Initialize();
    }
}

Playing Remote Audio

To play audio from a remote video feed, you must implement two methods. The first method, CreateAudioSink, returns an audio sink that will be used for audio playback. Normally, the audio sink is the system speaker. LiveSwitch provides a library for your platform that allows you to play audio using the user's speaker:

  • For Xamarin iOS: include FM.LiveSwitch.Cocoa.dll.
public class RemoteMedia : FM.LiveSwitch.RtcRemoteMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.AudioSink CreateAudioSink(FM.LiveSwitch.AudioConfig config)
    {
        return new FM.LiveSwitch.Cocoa.AudioUnitSink(config);
    }
}

The second method to implement is the CreateOpusDecoder method. The code for CreateOpusDecoder will be similar to the code for CreateOpusEncoder. Instead of supplying an encoder implementation, however, you supply a decoder implementation. Once complete you will be able to receive and decode audio sent from remote users. As you will already know from implementing the App.LocalMedia class, LiveSwitch provides an Opus library for your platform. If you haven't already, you must include it now:

  • For Xamarin iOS: include FM.LiveSwitch.Opus.dll.
public class RemoteMedia : FM.LiveSwitch.RtcRemoteMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.AudioDecoder CreateOpusDecoder(FM.LiveSwitch.AudioConfig config)
    {
        return new FM.LiveSwitch.Opus.Decoder(config);
    }
}

Playing Remote Video

To play remote video with the App.RemoteMedia class, you must implement several methods, in the same way that you had to implement several methods to capture local video with your App.LocalMedia class. The first method you must implement is CreateViewSink. This method creates a view object that will play back a remote video feed. LiveSwitch provides an implementation for your platform:

  • For Xamarin iOS: include FM.LiveSwitch.Cocoa.dll.
public class RemoteMedia : FM.LiveSwitch.RtcRemoteMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.ViewSink<FM.LiveSwitch.Cocoa.OpenGLView> CreateViewSink
    {
        return new FM.LiveSwitch.Cocoa.OpenGLSink();
    }
}

The next step is to implement the factory methods for the various video codec decoders. These are CreateVp8DecoderCreateVp9Decoder and CreateH264Decoder. For each method, create and return an instance of the appropriate decoder. As with encoders, if you do not wish to support a codec, you can return null. The implementation for your platform can be found in:

  • For Xamarin iOS: include FM.LiveSwitch.Vpx.dll.
  • Note that for Mac platforms, there is no OpenH264 library, as these platforms natively support the H264 codec.
public class RemoteMedia : FM.LiveSwitch.RtcRemoteMedia<FM.LiveSwitch.Cocoa.OpenGLView>
{
    public override FM.LiveSwitch.VideoDecoder CreateVp8Decoder()
    {
        return new FM.LiveSwitch.Vp8.Decoder();
    }
    public override FM.LiveSwitch.VideoDecoder CreateVp9Decoder()
    {
        return new FM.LiveSwitch.Vp9.Decoder();
    }
    public override FM.LiveSwitch.VideoDecoder CreateH264Decoder()
    {
        return new FM.LiveSwitch.H264.Decoder();
    }
}

Finally, you will need to implement the CreateImageConverter method. The implementation is identical to the implementation for your App.LocalMedia class. Simply return an instance of FM.LiveSwitch.Yuv.ImageConverter. If you have not included the libyuv library yet, make sure you do so:

  • For Xamarin iOS: include FM.LiveSwitch.Yuv.dll.
public class RemoteMedia : FM.LiveSwitch.RtcRemotelMedia<...>
{
    public override FM.LiveSwitch.VideoPipe CreateImageConverter(FM.LiveSwitch.VideoFormat outputFormat)
    {
        return new FM.LiveSwitch.Yuv.ImageConverter(outputFormat);
    }
}

Enabling Acoustic Echo Cancellation

If you have ever hear yourself speaking a few hundred milliseconds after you have spoken, this is most likely because the remote peer does not have Acoustic Echo Cancellation (AEC). The remote peer is playing your audio stream through the speakers, then picking it up with the microphone and streaming it back to you. Acoustic Echo Cancellation solves this by keeping track of the audio data you send. If someone retransmits data that is similar enough to the data you have just sent, it is recognized as an echo and is removed from the audio data before playback.

The good news is that iOS already provides echo cancellation. It is built right in and requires no action from you as a developer. Because Apple implements their own echo cancellation, there is no need for LiveSwitch to re-invent the wheel, so there are no AEC libraries for iOS.

Creating Streams and Connections

Creating Streams

You've learned that local media captures the audio and video data of the current user, and remote media displays the audio and video data of other users. What's needed now is some way to tie these together. The object that does this is known as a stream. A stream is a relationship between two media objects that defines how these objects will communicate with each other. This section will focus on how to use the App.LocalMedia and App.RemoteMedia classes that you have created to establish this communication.

Creating bi-directional streams

The most common type of stream is one that both sends and receives data. This is known as a bi-directional stream, because media data moves in two directions. To stream audio/video data, you must create two streams, one for audio data and one for video data. These streams are represented by the FM.LiveSwitch.AudioStream and FM.LiveSwitch.VideoStream classes. To create bi-directional audio and video streams, create an instance of each of these classes, and provide both your App.LocalMedia instance and an instance of your App.RemoteMedia class.

Note that you will only ever have one instance of your App.LocalMedia class. You will create this instance once, and you will invoke Start on this instance once. You will, however, have multiple instances of your App.RemoteMediaclass. You will need one remote media instance for each participant that joins a session. In turn, because a stream defines a relationship between two points, you will need one instance of AudioStream and one instance of VideoStream per user that joins a video conference.

The code sample below shows how you would establish a single connection with a user.

var remoteMedia = new RemoteMedia();
var audioStream = new FM.LiveSwitch.AudioStream(localMedia, remoteMedia);
var videoStream = new FM.LiveSwitch.VideoStream(localMedia, remoteMedia);

Creating uni-directional streams

You can also create a uni-directional stream. This is a stream in which data is either only sent or only received. If you pass null instead of a LocalMedia instance to your audio or video stream's constructor, then no data will be sent through the stream. This is known as a receive-only stream, demonstrated below:

var remoteMedia = new RemoteMedia();
var audioStream = new FM.LiveSwitch.AudioStream(null, remoteMedia);
var videoStream = new FM.LiveSwitch.VideoStream(null, remoteMedia);

The opposite of this is a send-only stream. In this case, you pass null instead of a RemoteMedia instance (see the example below). You now have local media and remote media, which control how data is captured and displayed. You also have streams that control how data is sent and received. What you need now is something to tie this all together. The final section talks about how to do this by creating a connection. We will cover this next.

var audioStream = new FM.LiveSwitch.AudioStream(localMedia, null);
var videoStream = new FM.LiveSwitch.VideoStream(localMedia, null);

Creating a Connection

connection is an object that communicates between two endpoints using the ICE (Interactive Connectivity Establishment) protocol. ICE itself is beyond the scope of this guide. All that you need to know for now is that a connection is responsible for establishing and maintaining communication between two parties in a video conference.

LiveSwitch actually has three types of connections, each of which is covered in its own section of the Getting Started guide. For the purposes of this specific section, you will only look at creating an MCU connection. See the next sectionfor more information on MCU connections.

At this point, you should have an FM.LiveSwitch.Channel instance that you obtained either during registration or by joining a channel afterwards. To create an McuConnection instance, invoke the CreateMcuConnection method of the channel, as shown below, and pass in your AudioStream and VideoStream instances.

var connection = channel.CreateMcuConnection(audioStream, videoStream);

The same procedure works for peer connections (using CreatePeerConnection) and SFU connections (using CreateSfuUpstreamConnection or CreateSfuDownstreamConnection).

MCU connections and peer connections can be bi-directional (send/receive) or uni-directional (send-only or receive-only). SFU connections are always uni-directional. SFU upstream connections are always send-only (local media only) and SFU downstream connections are always receive-only (remote media only).

If you are creating peer connections, you must specify one or more ICE servers to guarantee connectivity across networks. There are two types of ICE servers - STUN and STUN/TURN (generally referred to as just TURN). A STUN-only server allows clients that are behind a NAT to discover their public IP address, which remote clients need in order to establish a connection. A TURN server does this too, but is also capable of serving as a simple relay for traffic between two participants that are behind especially restrictive firewalls.

To specify an ICE server, create an instance of FM.LiveSwitch.IceServer. If you are specifying a STUN server, then you only need to specify a single parameter, the URI of the STUN server. You should specify the scheme, domain and port. For STUN servers, the scheme should be stun. For TURN servers, you must specify two additional parameters, the username and password to access the TURN server. For TURN IceServer instances, the scheme should be turn.

The code samples below show how to configure two ICE servers, one for STUN and one for TURN.

peerConnection.IceServers = new[]
{
    new FM.LiveSwitch.IceServer("stun:stun.liveswitch.fm:3478",
    new FM.LiveSwitch.IceServer("turn:turn.liveswitch.fm:3478", "test", "pa55w0rd!")
}

Opening an SFU Connection

SFU is an acronym that stands for Selective Forwarding Unit. An SFU is an endpoint in a media session that enhances the scalability of video conferencing sessions by forwarding audio and video data that it receives from connected users. In an SFU configuration, each user only has one upstream connection to the server, substantially reducing the amount of upload bandwidth required to run a video conference. Running LiveSwitch in SFU mode consumes more bandwidth on the server than if you were only using peer to peer connections but it also uses far less CPU than running LiveSwitch in MCU mode. Because of this, an SFU is a good middle-ground scaling option when you have an excess of bandwidth and a limited amount of CPU power.

Creating an SFU Connection

You must be registered with the LiveSwitch gateway and joined to a FM.LiveSwitch.Channel instance as well. This is covered in the previous Registering a Client section. You should also review the Handling Local Media, Handling Remote Media, and Creating Streams and Connections sections, which review the media components and streams that are used in this guide.

When creating an SFU session, there are two different types of connections to consider. The first type of connection is known as the upstream connection. This connection forwards your own audio and video data to the server, which will then forward it to other participants. Each participant in an SFU session will have one upstream connection. The second type of connection is known as a downstream connection. These connections receive audio and video data that is forwarded from the server. Each participant will have one downstream connection for every other participant in the video conference. This guide will first describe how to create an upstream connection and will then detail how to create downstream connections for other participants.

Making an Upstream Connection

To create your upstream connection, invoke the CreateSfuUpstreamConnection method of your Channel instance. Note that you must specify an audio and video stream for this method. This, in itself, is not unusual - but you must create the audio and video streams as send-only. To do so, when you create the FM.LiveSwitch.AudioStream and FM.LiveSwitch.VideoStream instances, do not specify a RemoteMedia instance. Instead, use the constructor overload that takes only a single LocalMedia instance.

This will return an FM.LiveSwitch.SfuUpstreamConnection instance that is set up to send, but not receive any data. Once you have created this connection, assign it ICE servers as you normally would and then invoke the Openmethod of the connection instance. The Open method returns a promise, which you should inspect to verify that your upstream connection is established successfully. This is demonstrated below.

var audioStream = new FM.LiveSwitch.AudioStream(localMedia, null);
var videoStream = new FM.LiveSwitch.VideoStream(localMedia, null);
var connection = channel.CreateSfuUpstreamConnection(audioStream, videoStream);
connection.IceServers = ...
connection.Open().Then((result) =>
{
    Console.WriteLine("upstream connection established");
}).Fail((Exception ex) =>
{
    Console.WriteLine("an error occurred");
});

You now have an upstream connection and can send data to the server, which will then be forwarded on to other participants. However, you now need to create a downstream connection for other participants that join the SFU session. The next section will show you how to do this.

Making Downstream Connections

In an SFU session, you must create a new downstream connection every time that a peer joins the channel. You do this by adding an event handler for the OnRemoteUpstreamConnectionOpen event. This event is raised whenever a remote user opens an upstream connection on this channel. The event handler has two tasks:

  • add a UI element for the downstream connection
  • create and open a downstream connection

The first thing you'll need to do in this event handler is create a RemoteMedia instance and add its view to the layout manager. You can add it by invoking the AddRemoteViewmethod of your LayoutManager instance. You will need to specify both an id and a view instance.

Start by updating the UI. Create a new instance of your RemoteMedia class. Next, add the RemoteMedia instance's associated view to your layout manager by invoking its AddRemoteView method. This method requires an id parameter and a view object, both of which can be accessed through properties on the RemoteMedia instance.

channel.OnRemoteUpstreamConnectionOpen += (remoteConnectionInfo) =>
{
    var remoteMedia = new RemoteMedia();
    layoutManager.AddRemoteView(remoteMedia.Id, remoteMedia.View);
    ...
};

Next, you must create an FM.LiveSwitch.SfuDownstreamConnection instance. You do this by invoking the CreateSfuDownstreamConnection method of your Channel instance. You must pass an instance of FM.LiveSwitch.ConnectionInfo into this method as the first parameter. An instance of this is available to you as one of the parameters of your event handler. Assign ICE servers to your newly-created SfuDownstreamConnectioninstance and then invoke its Open method. This will return a promise, the result of which you should inspect to ensure that the downstream connection is established properly. Once complete, your application will now successfully be able to manage both upstream and downstream portions of an SFU session. The next section will focus on how to properly teardown the session.

channel.OnRemoteUpstreamConnectionOpen += (FM.LiveSwitch.ConnectionInfo remoteConnectionInfo) =>
{
    ...
    var audioStream = new FM.LiveSwitch.AudioStream(null, remoteMedia);
    var videoStream = new FM.LiveSwitch.VideoStream(null, remoteMedia);
    var connection = channel.CreateSfuDownstreamConnection(remoteConnectionInfo, audioStream, videoStream);
    connection.IceServers = ...
    connection.Open().Then((object result) =>
    {
        Console.WriteLine("downstream connection established");
    }.Fail((Exception ex) =>
    {
        Console.WriteLine("an error occurred");
    });
};

Teardown

When a user leaves a session, you should remove the remote view associated with them. If you do not do so, the view will remain frozen on screen indefinitely. To remove a view when a user leaves, you must be notified when a peer's upstream connection closes. This is done by adding an OnStateChange event handler to each SfuDownstreamConnection instance. In this handler, you inspect the state of the SfuDownstreamConnection instance. If the state is Closing or Failing, you remove the associated remote view by invoking the RemoveRemoteView instance of your layout manager.

You should only add this handler to the downstream connections that you create, as upstream connections do not have a remote view associated with them.

channel.OnRemoteUpstreamConnectionOpen += (FM.LiveSwitch.ConnectionInfo remoteConnectionInfo) =>
{
    ...
    connection.OnStateChange += (FM.LiveSwitch.ManagedConnection c) =>
    {
        if (c.State == FM.LiveSwitch.ConnectionState.Closing || c.State == FM.LiveSwitch.ConnectionState.Failing)
        {
            layoutManager.RemoveRemoteView(remoteMedia.Id);
        }
    }
};

If you need to close a connection manually, you can do so by invoking the Close method of an SfuDownstreamConnection instance or SfuUpstreamConnection instance. The Close method returns a promise, the result of which you can inspect to ensure the connection has been closed properly. Note that if you close your upstream connection, that you will effectively disconnect from the session, because the other peers will see their downstream connections from you as dropped.

Closing a connection is straightforward - note though, that the result object in this promise is not used, which is why it is assigned the generic object type.

connection.Close().Then((object result) =>
{
    Console.WriteLine("connection closed");
}).Fail((Exception ex) =>
{
    Console.WriteLine("an error occurred");
});

You now know how to establish and tear down an SFU session. These sessions impose some extra bandwidth requirements on your server components but they allow you to significantly scale up the number of users in a session, due to lower upstream bandwidth requirements. If your server component cannot support the CPU-intensive requirements of an MCU, then an SFU is a good compromise that can still allow you to scale up noticeably.

Opening an MCU Connection

MCU is an acronym that stands for Multipoint Control Unit. An MCU is an endpoint in a media session that allows for large amounts of users to participate in a video conferencing session by mixing their audio and video streams together on the server. Each participant in a video conference connects directly to the MCU, and the MCU creates one combined audio and video stream, which it then sends back to everyone. For session participants, this mode uses the least amount of bandwidth and CPU, as each participant only has one upstream and one downstream connection. However, the downside of this is that this mode uses the largest amount of bandwidth and CPU resources on the server side, as the MCU is now responsible for performing all audio and video mixing.

Creating an MCU Connection

Start by registering with the LiveSwitch gateway and obtaining an FM.LiveSwitch.Channel instance. This is covered in the previous Registering a Client section. You should also review the Handling Local Media, Handling Remote Media, and Creating Streams and Connections sections, which review the media components and streams that are used in this guide.

In some ways, creating an MCU connection is the simplest option, because there is only one connection for you to manage. In other ways, however, it is trickier, because the output of a video conferencing session is completely controlled on the server. This guide will walk you through connecting to a LiveSwitch server in MCU mode, and will help you navigate some of the gotchas involved in doing so.

Making a Mixed Connection

To create your mixed connection, invoke the CreateMcuConnection method of your Channel instance, and pass in your audio and video streams. This will return an FM.LiveSwitch.McuConnection instance. With this McuConnection instance, you should assign ICE servers as described in the Creating Streams and Connections section. Once you have set these servers and any other connection-specific properties, open the connection by invoking the Open method. This method returns a promise, which resolves when the connection is established and is rejected if the connection does not. The code samples below show how to accomplish this.

Once complete, you will now be connected to the server with an MCU connection. Because this is an MCU connection, there are no other connections to establish. The only other time that you will have to deal with your McuConnection object is when you leave the session, which is covered in the next section.

var remoteMedia = new RemoteMedia();
var audioStream = new FM.LiveSwitch.AudioStream(localMedia, remoteMedia);
var videoStream = new FM.LiveSwitch.VideoStream(localMedia, remoteMedia);
var connection = channel.CreateMcuConnection(audioStream, videoStream);
layoutManager.AddRemoteView(remoteMedia.Id, remoteMedia.View);
connection.OnStateChange += (FM.LiveSwitch.ManagedConnection c) =>
{
    if (c.State == FM.LiveSwitch.ConnectionState.Closing || c.State == FM.LiveSwitch.ConnectionState.Failing)
    {
        layoutManager.RemoveRemoteView(remoteMedia.Id);
    }
};
connection.IceServers = ...
connection.Open().Then((result) =>
{
    Console.WriteLine("mixed connection established");
}).Fail((ex) =>
{
    Console.WriteLine("an error occurred");
});

Closing a Connection

You should close an MCU connection when you wish to leave a session. This is not strictly required, as LiveSwitch will detect when a user has disconnected. However, it is generally a better experience for the end-users if you explicitly destroy the connection yourself because the server is notified immediately that someone has left the session.

To close an MCU connection is, invoke the Close method of the FM.LiveSwitch.McuConnection instance. Similar to the Open method, Close returns a promise. Again, you should inspect the result of the promise to ensure that teardown is performed successfully. This is demonstrated below.

When you have finished working through this code example, you will have successfully opened and closed a connection to the server. However, you haven't really done anything with this connection yet. The next section of this guide focuses on actually displaying the data you receive from the MCU.

connection.Close().Then((object result) =>
{
    Console.WriteLine("connection closed");
}).Fail((Exception ex) =>
{
    Console.WriteLine("an error occurred");
});

Managing the Layout

There are two main aspects of layout that you have to consider - the local view and remote views. The local view is a video preview of your own camera. This is common in video conferencing applications, as users often want to adjust their camera so that they are properly in view. Remote views, on the other hand, are the views that show everyone else. In the context of an MCU, there is only one remote view - the remote view that displays the video stream generated by the MCU. The next sections will focus on managing the layout for this local and remote view.

Updating the MCU Layout

The number of participants in a media session often changes over time. As this occurs, the MCU automatically changes how it lays out the video feeds. Sometimes, this also requires an update on the client side, in order to display the layout with proper margins and padding. When such an update is required, an OnMcuVideoLayout event will be raised on the FM.LiveSwitch.Channel instance that is associated the MCU session. In your application, you must add an event handler to the Channel instance to handle this.

This event handler has two responsibilities. Its first responsibility is to call the Layout method of your layout manager. As a UI operation, this must take place on the main thread. The event handler's other responsibility is to cache the FM.LiveSwitch.VideoLayout instance that is returned as a parameter from the event handler. The use of the VideoLayout instance is covered in the next section - for now, know that you must store it somewhere.

channel.OnMcuVideoLayout += (FM.LiveSwitch.VideoLayout videoLayout) =>
{
    this.videoLayout = videoLayout;
 
    if (layoutManager != null)
    {
        Dispatcher.invoke(new Action(() =>
        {
            layoutManager.Layout();
        }));
    }   
};

The above code takes a new layout from the server and applies it to your layout manager. You may also wish to apply additional effects to the layout now, which is why you will need the previously cached VideoLayout instance. One common effect is to float the local video preview, so that it appears off to the side, above the remote video feeds. To do this, first add an OnLayout event handler to your FM.LiveSwitch.LayoutManager instance. In this event handler, invoke the FloatLocalPreview method of the FM.LiveSwitch.LayoutUtility utility class. You must pass in the previously cached VideoLayout instance, the Layout instance that you are provided as a parameter for the OnLayoutevent handler and the id of your McuConnection instance. The example below shows how to do this.

layoutManager.OnLayout += (FM.LiveSwitch.Layout layout) =>
{
    if (this.mcuConnection != null)
    {
        FM.LiveSwitch.LayoutUtility.FloatLocalPreview(layout, this.videoLayout, this.mcuConnection.Id);
    }
};

You now know how to setup and teardown an MCU connection. As mentioned above, this connection method is the simplest from an application perspective, but is most reliant on the bandwidth and CPU capabilities of your server. In general, you want to save MCU mode for conferencing sessions that truly require it - conferences of 9 or more people, as this can provide you substantial savings on server costs.

SFU/MCU Broadcasting

LiveSwitch supports massive-scale broadcasting of audio/video data from both SFU and MCU upstream connections to SFU downstream connections. Two call-flow models are available to support different use cases: Reactive Downstreams and Proactive Downstreams.

Reactive Downstream Connections

The reactive model is the easiest to implement, and is generally preferable for small audiences (less than 1,000). In this model, SFU downstreams are created from the client application when the client application receives an upstream notification.

_Channel.OnRemoteUpstreamConnectionOpen += (remoteConnectionInfo) =>
{
    SfuDownstreamConnection downstream = _Channel.CreateSfuDownstreamConnection(remoteConnectionInfo, ...);
};

Reactive SFU downstream connections are created by passing an instance of RemoteConnectionInfo into the CreateSfuDownstreamConnection channel method. The remote upstream connection can be any SFU or MCU connection in the same channel with an upstream (sending) component.

SfuUpstreamConnection upstream; // or MCU
upstream = _Channel.CreateSfuUpstreamConnection(...);

The lifespan of an SFU downstream connection created this way is tied to the remote upstream connection. When the remote upstream connection closes (or fails), the SFU downstream connection will also close.

Reactive Connection Bursts

A sudden flood of downstream connection requests will be generated if a large audience is present and actively waiting when the SFU or MCU upstream connection is created, or if the upstream connection loses network connectivity and has to create a new connection. LiveSwitch can handle this burst traffic, provided sufficient server resources are available, but consider the proactive model if you anticipate large audiences.


Proactive Downstream Connections

The proactive model is new in LiveSwitch 1.2.0 and is preferable for large audiences. In this model, SFU downstreams are created and managed from the client application independent of any remote upstream connection events. A media identifier, unique to a given channel, is used to associate an upstream connection with any downstream connections.

SfuDownstreamConnection downstream = _Channel.CreateSfuDownstreamConnection(“upstream media ID”, ...);

Proactive SFU downstream connections are created by passing a media identifier into the CreateSfuDownstreamConnection channel method. The remote upstream connection must use the same media identifier, but can otherwise be any SFU or MCU connection in the same channel with an upstream (sending component).

SfuUpstreamConnection upstream; // or MCU
upstream = _Channel.CreateSfuUpstreamConnection(..., "upstream media ID");


Upstream Notifications

The client application will still receive notifications when the remote upstream connections open and close (`OnRemoteUpstreamConnectionOpen` and `OnRemoteUpstreamConnectionClose`). These events can be used to update the UI as desired in response to upstream connection state changes.

Opening a P2P Connection

In addition to SFU and MCU connections, LiveSwitch also supports a peer-to-peer (P2P) connections. This allows for users to be connected to each other in a mesh topology, where each user is connected to every other user directly. This type of connection consumes the least amount of server resources but is also the most bandwidth and CPU-intensive type of connection for end-users. The amount of bandwidth required to maintain the session increases by a constant factor as each additional user joins. This makes peer to peer connections preferable for small video conferences, as users' connections can generally not support more than five or six people in a conference.

Creating a Peer to Peer Connection

Start by registering with the LiveSwitch gateway and obtaining an FM.LiveSwitch.Channel instance. This is covered in the previous Registering a Client section. You should also review the Handling Local Media, Handling Remote Media, and Creating Streams and Connections sections, which review the media components and streams that are used in this guide.

When establishing a peer to peer connection, there are two roles to consider. The first role is the offerer. The offerer is a user who is already in a channel. When a new user joins this channel, it is the offerer's responsibility to send this new user a message that indicates that they wish to connect to them. This leads to the second role - the answerer. The answerer is a user who has just joined a channel. They are responsible for answering any connection messages that they receive. This section will focus first on defining the offerer's role and then the answerer's role.

Defining the Offerer

To send an offer to clients who have just joined, you first need to be notified when a client joins. This is accomplished by adding an OnRemoteClientJoin event handler to your Channel instance. As indicated by the event name, this event is raised whenever a remote client joins the channel. This event handler must do two things:

  • update the UI
  • create and open a peer to peer connection

Begin by updating the UI. Create a new instance of your RemoteMedia class. Next, add the RemoteMedia instance's associated view to your layout manager by invoking its AddRemoteView method. This method requires an id parameter and a view object, both of which can be accessed through properties on the RemoteMedia instance. Next, you must create an FM.LiveSwitch.PeerConnection instance. You do this by invoking the CreatePeerConnection method of your Channel instance. This method expects an instance of FM.LiveSwitch.ClientInfo as its first parameter, which is provided to you as one of the event handler parameters. Create the PeerConnection instance, assign it the required ICE servers and kick off the connection process by invoking its Open method. This will send an offer to the user who has just connected, identified by the information in the ClientInfo instance. Remember to inspect the promise returned from the Open method.

Once complete, your application will be able to send offers to new users who join the channel. The next step is to have these users respond to the offers and establish a connection.

channel.OnRemoteClientJoin += (FM.LiveSwitch.ClientInfo remoteClientInfo) =>
{
    var remoteMedia = new RemoteMedia();
    var audioStream = new FM.LiveSwitch.AudioStream(localMedia, remoteMedia);
    var videoStream = new FM.LiveSwitch.VideoStream(localMedia, remoteMedia);
    var connection = channel.CreatePeerConnection(remoteClientInfo, audioStream, videoStream);
    layoutManager.AddRemoteView(remoteMedia.Id, remoteMedia.View);    connection.OnStateChange += (FM.LiveSwitch.ManagedConnection c) =>
    {
        if (c.State == FM.LiveSwitch.ConnectionState.Closing || c.State == FM.LiveSwitch.ConnectionState.Failing)
        {
            layoutManager.RemoveRemoteView(remoteMedia.Id);
        }
    }
    connection.IceServers = ...
    connection.Open().Then((result) =>
    {
        Console.WriteLine("offerer's connection established");
    }.Fail((ex) =>
    {
        Console.WriteLine("an error occurred");
    });
};

Defining the Answerer

The code for responding to an offer is almost identical to the code for sending one. The key difference is that instead of adding an OnRemoteClientJoin event handler, you will add an OnPeerConnectionOffer event handler. Similar to above, this event handler has two key responsibilities:

  • update the UI
  • create and open a peer to peer connection

Proceed as you did above when writing this event handler. Create a RemoteMedia instance, and add the new view to the layout. Once again, invoke CreatePeerConnection, but this time you will pass it an instance of FM.LiveSwitch.PeerConnectionOffer. You can obtain this instance from the parameters of the event handler. The CreatePeerConnection method recognizes that because you have provided it with an offer, that it must now respond with an answer. Invoke the Open method of the created FM.LiveSwitch.PeerConnection instance and an answer will be generated and sent back to the original peer. As usual, the Open method returns a promise wrapping the result of the Open invocation.

Once you have worked through the example below you will have now defined both the offerer and answerer role, and two or more users should be able to connect to each other when they join a channel. The next sections will focus on what to do when users leave and how to properly end a peer to peer session.

channel.OnPeerConnectionOffer += (peerConnectionOffer) =>
{
    var remoteMedia = new RemoteMedia();
    var audioStream = new FM.LiveSwitch.AudioStream(localMedia, remoteMedia);
    var videoStream = new FM.LiveSwitch.VideoStream(localMedia, remoteMedia);
    var connection = channel.CreatePeerConnection(peerConnectionOffer, audioStream, videoStream);
    layoutManager.AddRemoteView(remoteMedia.Id, remoteMedia.View);
    connection.OnStateChange += (FM.LiveSwitch.ManagedConnection c) =>
    {
        if (c.State == FM.LiveSwitch.ConnectionState.Closing || c.State == FM.LiveSwitch.ConnectionState.Failing)
        {
            layoutManager.RemoveRemoteView(remoteMedia.Id);
        }
    }
    connection.IceServers = ...
    connection.Open().Then((object result) =>
    {
        Console.WriteLine("answerer's connection established");
    }.Fail((Exception ex) =>
    {
        Console.WriteLine("an error occurred");
    });
};

Closing a Connection

If you need to close a connection manually, you can do so by invoking the Close method of the PeerConnection instance. The Close method returns a promise, the result of which you can inspect to ensure the connection has been closed properly.

Closing a connection is straightforward - note though, that the result object in this promise is not used, which is why it is assigned the generic object type.

connection.Close().Then((object result) =>
{
    Console.WriteLine("connection closed");
}).Fail((Exception ex) =>
{
    Console.WriteLine("an error occurred");
});

You now know how to properly establish and tear down a peer to peer session. Once again, peer to peer sessions work best with small amounts of participants, as the required bandwidth escalates quickly as more people join the session. You should still try to use them whenever possible, however, as they can provide substantial savings on server bandwidth.

Supporting H.264

The two video codecs supported by LiveSwitch are VP8 and H.264. All LiveSwitch SDKs support VP8 out-of-the-box, and almost all LiveSwitch SDKs support H.264 out-of-the-box too. Unfortunately, the Xamarin Cocoa platforms are the exception. These platforms will not support H.264 until the Cocoa VideoToolbox is ported to Xamarin Cocoa.

Platform Support for H.264

Web

Browsers supply their own codecs internally. You do not need to do anything special to support H.264 in Web clients. It depends solely on whether the browser supports the codec.

Chrome Firefox Edge (40+) IE11 Safari Chrome on Android Chrome on iOS Safari on iOS Firefox on Android Firefox on iOS Opera
Yes Yes Yes No Yes No rcvonly Yes Yes rcvonly Yes
  • rcvonly - getUserMedia is not supported by the browser, so while the codecs are supported, receive only streams are possible whereas send/receive or send only streams are not supported.

Native Platforms

.NET UWP Java Android iOS Obj-C iOS Swift macOS Obj-C macOS Swift Xamarin Android Xamarin iOS Xamarin macOS
Yes Partial1 Yes Partial2 Partial3 Partial3 Partial3 Partial3 Partial2 No No
  1. Very partial. An OpenH264 integration library is included, but the OpenH264 library cannot currently be downloaded at runtime due to UWP restrictions. To support H.264 for UWP you would have to include the OpenH264 binary provided by Cisco into your release and pay the corresponding royalties to MPEG LA (http://www.mpegla.com/main/programs/AVC/Pages/Intro.aspx).
  2. On Android, Cisco currently provides OpenH264 binaries for armeabi-v7a only.
  3. Both macOS and iOS support H.264 natively, but there are known issues with VideoToolbox on macOS and iOS that limit H.264 support on these platforms.

Media Servers

LiveSwitch Media Servers are already set up to support H.264 as per the best practices outlined here. Your Media Servers will download and install the OpenH264 binary automatically. The only action required to enable H.264 on your Media Server is to add the configuration option to turn the feature on. See the Modifying Codec Settings of the Configuring the Media Server documentation for more information.

SFU/MCU Transcoding

Transcoding Solves Codec Mismatch

Note that even for those clients that cannot support H.264, either natively or using OpenH264, if you are using MCU/SFU connections then transcoding solves this interoperability problem for you.

Media Servers automatically transcode video as necessary for SFU/MCU connections. For example, in an SFU connection if one peer negotiates H.264 up and down, and another negotiates VP8, then the Media Server will transcode the video from these peers to the appropriate encoding before forwarding it.

Starting a Screen Capture

In the section on Creating Streams and Connections, you learned how to capture input from a user's camera and microphone. You are not limited to a user's camera; you can use also use LiveSwitch to perform a screen share. This section will demonstrate how you can allow your users to capture their screen data and share it with others in a session.

Capturing a User's Screen

In other sections, you learned that to capture a user's microphone and camera, you must provide a class derived from the LiveSwitch LocalMedia class. In this class you had to override the CreateVideoSource function and with it create and return a CameraSource. If you want to support screen capture, then you need to do something similar, but instead of returning a CameraSource video source you will return a ScreenSource like in the code below:

iOS uses the FM.LiveSwitch.Cocoa.ScreenSource, which supports capturing the screen of your app only. This is due to restrictions that iOS places on the underlying API used by the source. We do have plans to provide a ReplayKit integrated source, which will support capturing the screen of the device in general, but this is unavailable at this time.

public class LocalScreenMedia : FM.LiveSwitch.RtcLocalMedia<FM.LiveSwitch.Cocoa.UIImageView>
{
    public override FM.LiveSwitch.VideoSource CreateVideoSource()
    {
        return new FM.LiveSwitch.Cocoa.ScreenSource(3);
    }


	protected override ViewSink<UIImageView> CreateViewSink()
    {
        return new ImageViewSink();
    }
}

To capture a user's screen instead of their camera, simply use this new LocalScreenMedia class instead of the LocalCameraMedia class. It's a simple as that!

Now you can establish a connection as you normally would, but instead of the camera, you will now be sharing the user's screen. But what if you want to share both the user's camera and their screen? This is a common scenario. The best way to address this is to first connect with everyone as you normally would. Then, when a user wishes to share their screen, create a new connection specifically for the screen share, and disable audio, like so:

bool DisableAudio = true;
bool DisableVideo = false;
var LocalScreenMedia = new LocalScreenMedia(DisableAudio, DisableVideo, ...);

Once you have the screen media, establish a connection with each user in the session to share your screen.

Best Practice

If you wish to share video from the camera and video from screen capture at the same time then the best practice is to create two connections to handle this scenario. One connection manages video from the camera (and audio if needed), and the other manages video from the screen capture (disable audio in this connection). This makes it trivial to end the screen share at any point and also allows users to selectively mute the screen share if it is using up too much bandwidth. Also, the browsers do not allow more than one video stream per connection, so if you want to support this use case, and interop with browser clients, then you must manage camera and screen concerns in separate connections.

You've now learned how to capture and share a user's screen. When sharing screens, remember that the bandwidth used to share a screen is several orders larger than the typical bandwidth used to share a camera. This is not usually a problem on modern networks, but make sure that you don't allow every user to share their screen at once to avoid reducing the video quality of the conference.

Making an Outbound SIP Call

The LiveSwitch SIP Connector allows your application to make calls to SIP endpoints. This allows you to connect WebRTC and SIP clients seamlessly in a single conference. Before continuing, make sure that you have installed and configured the SIP Connector. Refer to the section in the Server docs on Configuring the SIP Connector for information on how to do so.

Making an Outbound Call

To make an outbound call, first obtain a reference to the channel that you want to add the SIP user to. Invoke the Channel instance's Invite method and specify both the user id of the SIP endpoint that you are trying to reach and the protocol that you want to use. The protocol parameter must be either "tel" or "sip". These values are treated the same by LiveSwitch, so you can select either one.

The Invite method returns a promise, which resolves if the invitation is sent successfully, and is rejected if it is not sent. Note that it the resolve action of this promise does not mean that the remote SIP user has connected to the conference, only that the LiveSwitch SIP Connector has successfully received the invitation and forwarded it to the remote SIP user. If the remote party accepts the invitation, they will be added to the conference. An example of this is demonstrated below.

FM.LiveSwitch.Channel channel;
 
channel.Invite("1234", "tel").Then((FM.LiveSwitch.Invitation invitation) =>
{
    Console.WriteLine("sent SIP invitation");
}).Fail(ex =>
{
    Console.WriteLine("failed to send SIP invitation");
});

Cancelling a Call

The Invite method's promise object returns an FM.LiveSwitch.Invitation instance when successful. The main use for this object is to cancel the invitation. You can do so by invoking the Cancel method of the Invitationinstance. Note that once an outgoing call is answered, you can no longer cancel the invitation. An example of cancelling a call is shown below:

FM.LiveSwitch.Channel channel;
 
channel.Invite("1234", "tel").Then((FM.LiveSwitch.Invitation invitation) =>
{
    Console.WriteLine("sent SIP invitation");
 
    invitation.Cancel().Then(obj =>
    {
        Console.WriteLine("cancelled SIP invitation");
    }).Fail(ex =>
    {
        Console.WriteLine("failed to cancel SIP invitation");
    });
}).Fail(ex =>
{
    Console.WriteLine("failed to send SIP invitation");
});

Working with Dual-tone Multi-frequency (DTMF) Signalling

LiveSwitch 1.1.0 added support for dual-tone multi-frequency signalling (DTMF) for peer connections.

MCU

This feature is not yet supported for MCU connections.

Sending DTMF Tones

Here we demonstrate how simple it is to send a DTMF tone.

SFU streams

Note that SFU upstreams are send only by definition, so they can send DTMF tones, but not receive them.

audioStream.InsertDtmfTones(FM.LiveSwitch.Dtmf.Tone.FromToneString("..."));


DTMF Tone String

A tone string is made of up characters from the set "123456789*0#ABCD,". The special character "," indicates a pause.


Also, you can hook into an event that is raised when the sending tone changes.

audioStream.OnSendDtmfToneChange += (tone) =>
{
    Log.Info("Sender's DTMF tone is changing: " + tone.ToString());
};

The empty string “” indicates that the tone has stopped.

For browsers, that's all there is to it. Per Mozilla:

The primary purpose for WebRTC's DTMF support is to allow WebRTC-based communication clients to be connected to a public-switched telephone network (PSTN) or other legacy telephone service, including extant voice over IP (VoIP) services. For that reason, DTMF can't be used between two WebRTC-based devices, because there is no mechanism provided by WebRTC for receiving DTMF codes.

Additional DTMF Tone Features for Native Platforms

We've taken it a few steps further on all other platforms (native desktop/mobile), where additional sending and receiving events are available.

When the receiving tone changes, an event is raised:

audioStream.OnReceiveDtmfToneChange += (tone) =>
{
    Log.Info("Receiver's DTMF tone is changing: " + tone.ToString());
};

The empty string “” indicates that the tone has stopped.

If you want to go lower-level, it's possible to handle the actual packet-level send/receive events which are tied to the clock-rate of the selected audio stream codec (Opus, PCMU, PCMA, etc.).

SFU streams

Note that SFU upstreams are send only by definition, so they can send DTMF tones, but not receive them. Similarly, SFU downstreams are receive only by definition. so they can receive DTMF tones but cannot send them.

audioStream.OnSendDtmfTone += (tone) =>
{
    Log.Info("Sending DTMF tone: " + tone.ToString());
};
audioStream.OnReceiveDtmfTone += (tone) =>
{
    Log.Info("Received DTMF tone: " + tone.ToString());
};

Creating Custom Sources and Sinks

The LiveSwitch API uses the concepts of sources and sinks, which you should already be familiar with. To review briefly, a source receives data to send to another user and a sink displays data that was sent from another user. LiveSwitch includes several implementations of sources and sinks for common use cases. However, some application-specific use cases may require you to implement your own sources and sinks.

Some examples of use cases that LiveSwitch does not support by default: - using a video file from a user's phone as a source - streaming received video to other devices

Before you start working on your own source or sink, note that you should try not to implement sources or sinks whose only purpose is to apply transformations to a video or audio stream. These use cases are more easily dealt with by the media chaining API, which has a guide under the Advanced Topics section.

Prerequisites

Before working through this guide, ensure that you have a working knowledge of the following topics:

  • LiveSwitch Local Media API
  • LiveSwitch Remote Media API
  • LiveSwitch Streams API

These are covered earlier.

Audio and Video Formats

To implement a custom source or sink, some knowledge about the way LiveSwitch handles audio and video formats is required. Each FM.LiveSwitch.AudioSource and FM.LiveSwitch.AudioSink instance has an associated FM.LiveSwitch.AudioFormat instance. A format consists of a clock rate, in Hz and the number of audio channels. For a source, the format indicates the format of the audio output by the source. For a sink, the format indicates the format that the sink expects input audio to be in.

You will need to specify AudioFormat instances for your sources and sinks. There are a number of pre-defined formats you can use. The following code demonstrates the creation of several AudioFormat instances.

var opusFormat = new FM.LiveSwitch.Opus.Format();
var pcmaFormat = new FM.LiveSwitch.Pcma.Format();
var pcmuFormat = new FM.LiveSwitch.Pcmu.Format();

One other format implementation that may be useful is the FM.LiveSwitch.Pcm.Format class. This format allows you to specify a generic PCM format with a custom clock rate and audio channel count. The following code demonstrates creating a 48,000Hz, 2 channel audio format instance.

var pcmFormat = new FM.LiveSwitch.Pcm.Format(48000, 2);

For the complete list of pre-defined audio formats, refer to the API documentation.

Like audio sources and sink, each FM.LiveSwitch.VideoSource and FM.LiveSwitch.VideoSink instance also has an associated FM.LiveSwitch.VideoFormat instance. Video formats consist of a clock rate and information about the colorspace of the format. Again, like audio sources and sinks, there is a set of pre-defined video formats to select from. The following code demonstrates creating instances of two of the most common video formats, RGB and I420.

Refer to the API docs for the complete list of supported formats.

var rgbFormat = FM.LiveSwitch.VideoFormat.Rgb;
var i420Format = FM.LiveSwitch.VideoFormat.I420;

Custom Sources

To create a custom audio or video source, first inherit from either the FM.LiveSwitch.AudioSource or the FM.LiveSwitch.VideoSource class. Neither of these classes have a default constructor; they require you to specify either an FM.LiveSwitch.AudioFormat or an FM.LiveSwitch.VideoFormat instance. Most custom sources are designed for a specific output format, so it is common to create a default constructor that invokes the base constructor with a pre-defined format. The following code demonstrates this.

public class CustomAudioSource : FM.LiveSwitch.AudioSource
{
    public CustomAudioSource()
        : base(new FM.LiveSwitch.Pcm.Format(48000, 2))
    {
    }
}
 
public class CustomVideoSource : FM.LiveSwitch.VideoSource
{
    public CustomVideoSource()
        : base(FM.LiveSwitch.VideoFormat.Rgb)
    {
    }
}

Next, override the Label property. This is an accessor that returns a string that identifies the type of source. The value you provide here is only for diagnostic purposes and will not affect the output of an audio or video source.

public class CustomAudioSource : FM.LiveSwitch.AudioSource
{
    public override string Label => "CustomAudioSource";
}
 
public class CustomVideoSource : FM.LiveSwitch.VideoSource
{
    public override string Label => "CustomVideoSource";
}

Finally, you must implement the DoStart and DoStop methods. Usually, these methods will follow one of two patterns. They will either manage an event handler on a interface that captures audio and video data or they will manage a separate thread that runs in the background, which will generate audio and video data. With both patterns, the source must invoke the RaiseFrame method when data is available. RaiseFrame is a protected method that signals to components in the media stack that new data is available.

Note that the DoStart and DoStop methods are asynchronous and return an FM.LiveSwitch.Future. For the sake of simplicity, these examples are synchronous and will resolve the promise immediately. In practice, your implementation will likely be more complex.

Capturing Audio

The first set of examples demonstrate how to begin capturing audio using the event-based pattern. Do not use any of these snippets as-is. They are not full implementations. They are only designed to give a brief overview of how you might implement these patterns in your own application. The examples do not cover important details such as endianness or upsampling/downsampling of audio. These details are glossed over in the samples.

A fictional AudioCaptureObject class is used for these examples. When implementing your own source, you should adapt the source here to your own application. The example code first creates an instance of the AudioCaptureObject, then adds an event handler that will be raised whenever new audio data is available. Note that the event handler has two parameters, duration and data. The data parameters is a byte array containing raw audio data over a period of time. The duration parameter is the number of milliseconds that this data represents. You must calculate the duration yourself, as it will vary based on your specific implementation.

With these two parameters, you can finally raise an audio frame, though there are several intermediate steps. First, wrap the raw audio data in an instance of FM.LiveSwitch.DataBuffer. Next, wrap the data buffer in an instance of FM.LiveSwitch.AudioBuffer, which also requires you to specify the FM.LiveSwitch.AudioFormat of the audio data. You can simply use the OutputFormat property of your audio source to retrieve this. Finally, wrap the audio buffer in an instance of FM.LiveSwitch.AudioFrame and provide the audio duration. Invoke RaiseFrame on this new AudioFrame instance.

public class CustomAudioSource : FM.LiveSwitch.AudioSource
{
    private AudioCaptureObject _Capture;
 
    public override FM.LiveSwitch.Future<object> DoStart()
    {
        var promise = new FM.LiveSwitch.Promise<object>();
 
        _Capture = new AudioCaptureObject();
        _Capture.AudioDataAvailable += (double duration, byte[] data) =>
        {
            var dataBuffer = FM.LiveSwitch.DataBuffer.Wrap(data);
            var audioBuffer = new FM.LiveSwitch.AudioBuffer(dataBuffer, this.OutputFormat);
            var audioFrame = new FM.LiveSwitch.AudioFrame(duration, audioBuffer);
 
            this.RaiseFrame(audioFrame);
        });
 
        promise.resolve(null);       
        return promise;
    }
}

Stopping an FM.LiveSwitch.AudioSource instance is much simpler. Destroy whatever capture interface you were using or remove any event handlers.

public class CustomAudioSource : FM.LiveSwitch.AudioSource
{
    public override Future<object> DoStop()
    {
        var promise = new FM.LiveSwitch.Promise<object>();
 
        _Capture.Destroy();
        _Capture = null;
 
        promise.resolve(null);
        return promise;
    }
}

Capturing VIdeo

The next set of examples demonstrate how to capture video; again, using the event-based pattern. Like the previous examples, a fictional VideoCaptureObject class is used for these examples. The parameters associated with video data are slightly different than those associated with audio data. Instead of a duration, you need to specify the width and height of the video frames.

To raise a video frame, proceed the same way you did when raising an audio frame. First, wrap the raw video data in an instance of FM.LiveSwitch.DataBuffer. Next, wrap the data buffer in an instance of FM.LiveSwitch.VideoBuffer, which also requires you to specify the FM.LiveSwitch.VideoFormat of the data as well as the width and height of the video. Like the audio source, you can use the OutputFormat property of your video source. Finally, wrap the video buffer in an instance of FM.LiveSwitch.VideoFrame and invoke RaiseFrame.

public class CustomVideoSource : FM.LiveSwitch.VideoSource
{
    private VideoCaptureObject _Capture;
 
    public override FM.LiveSwitch.Future<object> DoStart()
    {
        var promise = new FM.LiveSwitch.Promise<object>();
 
        _Capture = new VideoCaptureObject();
        _Capture.VideoDataAvailable += (int width, int height, byte[] data) =>
        {
            var dataBuffer = FM.LiveSwitch.DataBuffer.Wrap(data);
            var videoBuffer = new FM.LiveSwitch.VideoBuffer(width, height, dataBuffer, this.OutputFormat);
            var videoFrame = new FM.LiveSwitch.VideoFrame(videoBuffer);
 
            this.RaiseFrame(videoFrame);
        });
 
        promise.resolve(null);       
        return promise;
    }
}

Stopping an FM.LiveSwitch.VideoSource is as simple as stopping an audio source - simply release or destroy any resources you were using. The code below will look very similar to what you've already written.

public class CustomVideoSource : FM.LiveSwitch.VideoSource
{
    public override Future<object> DoStop()
    {
        var promise = new FM.LiveSwitch.Promise<object>();
 
        _Capture.Destroy();
        _Capture = null;
 
        promise.resolve(null);
        return promise;
    }
}

Raising Frames

Raising frames was not covered in much detail above other than how to do it. One thing to keep in mind is that the rate at which you raise frames will determine the frame rate of your source. For audio, this has fewer implications because audio capture is associated with a duration. For video capture, it gets more complex.

Video buffers don't have a timestamp or duration associated with them. Regardless of whether you raise 15 video frames in a second or 30 frames, all of those video frames will be displayed during that second. If performance becomes an issue, you may need to throttle this method to limit the amount of frames that are raised per second.

Custom Sinks

Like you did when creating a custom source, to create a custom audio or video sink, first inherit from either the FM.LiveSwitch.AudioSink or the FM.LiveSwitch.VideoSink class. Sinks also require you to specify an FM.LiveSwitch.AudioFormat or an FM.LiveSwitch.VideoFormat instance. In this case, the format represents the input into the sink, rather than the output from the source. The following code is a simple example of how to create a custom sink. It should look familiar.

public class CustomAudioSink : FM.LiveSwitch.AudioSink
{
    public CustomAudioSink()
        : base(new FM.LiveSwitch.Pcm.Format(48000, 2))
    {
    }
}
 
public class CustomVideoSink : FM.LiveSwitch.VideoSink
{
    public CustomVideoSink()
        : base(FM.LiveSwitch.VideoFormat.RGB)
    {
    }
}

Sinks also have a Label property. Like the property of the same name on sources, it is used for diagnostic purposes and has no effect on what goes into your sinks.

public class CustomAudioSink : FM.LiveSwitch.AudioSink
{
    public override string Label => "CustomAudioSink";
}
 
public class CustomVideoSink : FM.LiveSwitch.VideoSink
{
    public override string Label => "CustomVideoSink";
}

At this point, the implementation for sinks is different. There are no DoStart or DoStop methods because sinks do not follow a "start/stop" pattern. Instead, whenever an audio or video frame is available, the sink will invoke its DoProcessFrame method. When a sink is instantiated, it is assumed to be ready to receive frames. The last method that sinks must implement is DoDestroy, which LiveSwitch invokes when tearing down a session. Its purpose is to clean up any resources that are still in use.

Unlike the audio and video source methods, DoProcessFrame and DoDestroy are synchronous, and do not return an FM.LiveSwitch.Promise. The reason they are synchronous is because they will never be invoked on the main thread. As a side effect of this, you must ensure that these methods are thread-safe.

Rendering Audio

This set of examples demonstrate how to play received audio data. It uses a fictional AudioRenderObject class, which abstracts away many of the details of audio playback. This does not mean that you can ignore them - in your own implementation, you must still deal with the upsampling and downsampling of audio.

There are many properties that are accessible from the FM.LiveSwitch.AudioFrame and FM.LiveSwitch.AudioBuffer classes. This example will focus on retrieving the two properties that were used in the sources example above: duration and data. Assume that the AudioRenderObject has a method, PlayAudio, that takes a duration parameter and a data parameter. You must retrieve these values from either the audio buffer or the audio frame.

First, retrieve the duration of the audio frame, by accessing the Duration properly of the AudioFrame parameter. Next, access the data buffer object by accessing the DataBuffer property of the AudioBuffer. With this DataBuffer instance, you can retrieve the raw audio data through the Data property. Finally, pass these values into the AudioRenderObject or whatever interface you are using for this sink.

public class CustomAudioSink : FM.LiveSwitch.AudioSink
{
    private AudioRenderObject _Render = new AudioRenderObject();
 
    public override void DoProcessFrame(FM.LiveSwitch.AudioFrame frame, FM.LiveSwitch.AudioBuffer buffer)
    {
        var duration = frame.Duration;
 
        var dataBuffer = buffer.DataBuffer;
        var data = dataBuffer.Data;
 
        _Render.PlayAudio(duration, data);
    }
}

For the implementation of the DoDestroy method, call any disposal methods and unset anything that is no longer in use.

public class CustomAudioSink : FM.LiveSwitch.AudioSink
{
    public override void DoDestroy()
    {
        _Render.Destroy();
        _Render = null;
    }
}

Rendering Video

Rendering video is similar to rendering audio. As we've been doing for the previous examples, we'll demonstrate video playback using a fictional VideoRenderObject class. The example will demonstrate how to retrieve the width and height of video data in the DoProcessFrame method, as well as the raw video data itself.

Retrieve the dimensions of the video by accessing the Width and Height properties of the video buffer parameter. Next, access the data buffer object by accessing the DataBuffer property of the same parameter. The raw video data is accessible through the Data property of the data buffer. You can pass these values into the AudioRenderObject or whatever interface you are using for this sink.

These examples will also cover a basic implementation of DoDestroy, which should look almost identical to that of those from the previous example set.

public class CustomVideoSink : FM.LiveSwitch.VideoSink
{
    private VideoRenderObject _Render = new VideoRenderObject();
 
    public override void DoProcessFrame(FM.LiveSwitch.VideoFrame frame, FM.LiveSwitch.VideoBuffer buffer)
    {
        var width = buffer.Width;
        var height = buffer.Height;
 
        var dataBuffer = buffer.DataBuffer;
        var data = dataBuffer.Data;
 
        _Render.PlayVideo(width, height, data);
    }
 
    public override void DoDestroy()
    {
        _Render.Destroy();
        _Render = null;
    }
}

Working with Data Channels

DataChannels support transfer of short message binary or text data. This could be useful for transferring any sort of data outside of the typical audio/video use case typically associated with RTC. LiveSwitch 1.2.0 introduces support for DataChannels for MCU and SFU connections - an industry first!

File Transfer

Our current implementation of DataChannels is optimized for small chunks of binary or text messages rather than for file transmission. While sending large files is possible, the rate will be restricted statically as we do not yet have full dynamic congestion control in place. At the moment, we are working on the Partial Reliability extension of SCTP (https://tools.ietf.org/html/rfc3758). Fast data transmission rates for large portions of data (such as for file transfer) will only be recommended after full implementation of congestion control.

Platform Support and Interoperability

In general, we work very hard to ensure full support of for features across all platforms. This is the case with DataChannels as well, but there is one exception: Microsoft Edge does not support DataChannels. For Web, LiveSwitch is limited by what features the browser provides access to, and in the case of Edge, we cannot offer support for DataChannels. 

This means that in your Web application code you must detect support for DataChannels, and where they are unsupported, forgo their creation and usage. The preferred way to do this would be feature detection. However, this is not advisable! Edge actually exposes the underlying API for DataChannel creation, so you can create DataChannels and a DataStream, add them to your Connection. Only once you attempt to send the SDP will you be able to detect that there is no SDP for the DataChannel, which of course is far to late in the process to make this useful. Feature detection tests in Edge (like detecting support for RTCDataChannel) will pass ... even though Edge will not give you a Connection with a DataChannel. So, in our Web example we use browser detection instead, and we disable all DataChannel related code in the case where we detect we are running on Edge. You should do something similar, either using the convenience browser detection offered by our Util class, or you can provide your own.

Detecting support
if (fm.liveswitch.Util.isEdge())
{
    // DataChannels are not supported ...
}

This of course, implies a connectivity problem. On the one had you may have a peer offering a DataStream as part of its SDP, and on the other a peer who is not. Traditionally this would result in an error, and the connection will not be established. LiveSwitch MCU and SFU connections do not suffer from this problem because the Media Server acts as a "middle man" for MCU and SFU connections. For DataChannel interoperability this means that the Media Server will successfully negotiate a DataStream for the peer that supports it, and also successfully negotiate no DataStream for the peer that does not. Obviously, data will not flow between the peers in this case, but the connection will be established, and audio/video will flow between the peers normally.

This table shows, per platform, connection types that will be successful between peers offering a DataStream, and Edge (not offering a DataStream).


Android

(including Xamarin)

iOS

(including Xamarin)

.NET Java mac OS UWP Chrome Firefox Safari IE Edge
Edge MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1 MCU/SFU1
IE MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P

Safari MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P
Firefox MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P
Chrome MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P
UWP MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P
mac OS MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P
Java MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P
.NET MCU/SFU/P2P MCU/SFU/P2P MCU/SFU/P2P
iOS
(including Xamarin)
MCU/SFU/P2P MCU/SFU/P2P

Android

(including Xamarin)

MCU/SFU/P2P
  1. MCU/SFU: Connections are successful regardless of platform, but Edge peers cannot send, and will not receive, data sent by others due to lack of DataChannel support on Edge. Audio/video will behave normally.

P2P Best Practice

P2P connection attempts to an Edge peer that include a DataStream will fail. To handle this case gracefully the best practice is to separate your DataStream into a separate P2P connection. That way you can depend on your P2P audio/video connection even where the DataStream connection is not possible.

Data Channel Usage

Adding DataChannel functionality is straight forward. You simply create the DataChannel, wrap it in a DataStream, and then provide that stream to your connection object in the same way you would do so for an AudioStream or a VideoStream. If required, you could have more than one DataChannel in your DataStream (you can have as many as needed, up to 65535). For example, perhaps you want one for chat messages, and another for sending spatial data. Whatever your use case, multi DataChannel solutions can help you keep your data concerns separate, which can in turn simplify your code. Of course, managing the life cycle of your DataChannels is something you need to manage in your application level code. Let's take a look at how to do just that. We'll start by creating a single DataChannel and adding it to a DataStream, which we then add to our connection.

Matching Channels

Please note that if you use multiple channels on the same stream, these channels are matched according to their labels by the peer. Therefore, you are not required to supply DataChannels in the same order for all peers. Instead, they are matched based on the channel label that was supplied at channel creation time.

For dynamically created channels, if one peer has not declare a channel on which another peer attempts to communicate, you can still receive data on that channel by subscribing to DataStream.OnChannel event, and then subscribing to the OnReceive event of that channel as described in the examples below.

DataChannel dataChannel = new DataChannel("my-data-channel");
DataStream dataStream = new DataStream(dataChannel);
McuConnection connection = _Channel.CreateMcuConnection(..., dataStream); // or SFU

Now that we know how to create our DataChannel, and add it to our Connection, let's take a look at how to manage the DataChannel throughout it's lifecycle. DataChannels have an associated state, much like connections, and it is up to your application level code to manage this state by hooking into state change events like this:

DataChannel dataChannel = new DataChannel("my-data-channel");
dataChannel.OnStateChange += (e) =>
{
    // States are New, Connecting, Connected, Closing, Closed, Failed
    if (e.State == DataChannelState.Connected)
    {
        ...
    }
    ...
};


DataChannelState.Connected

You can only actually send/receive data on a DataChannel that is in the DataChannelState.Connected state. The other states are handy for managing your DataChannel resources, but the Connected state tells you whether you can make use of the DataChannel.

Receiving data on your channel is also handled via hooking into an event. In this case we make use of the OnReceive event like this:

DataChannel dataChannel = new DataChannel("my-data-channel")
{
    OnReceive = (dataChannelReceiveArgs) => {
        if (dataChannelReceiveArgs.DataString != null)
        {
             ...
        }
    }
};

And last but not least, let's look at how to send on the DataChannel. Remember, it is important to ensure that you are not trying to send unless the DataChannel is in the connected state:

if (dataChannel.State == DataChannelState.Connected)
{
    dataChannel.SendDataString("Hello world!");
}

Connection Statistics

The stats API makes it easy to get information about your ongoing connections and other WebRTC internals. It closely follows the WebRTC Statistics API RFC. Note that the underlying WebRTC statistics API is not supported consistently from browser to browser. For instance Chrome exposes the most full-featured statistics whereas Edge does not implement it at all. In all cases where our statistics API is wrapping that provided by the browser, if the browser does not implement the given API then we simply return a null stats object. This is the expected behaviour and application code should be checking for null stats objects.

You can call the getStats function on any active Connection. For example, one way to do so would be to wire up an event to call getStats once the Connection state transitions to Connected, and then unwire the event when the Connection state transitions to Closed (or Failed). Your event could be triggered by some UI element (like a button click), or fired on a Timer, whatever meets your use case.

What follows is a code snippet showing how to get connection statistics:

getStats
connection.GetStats().Then((stats) => {
	var transport = stats.Streams[0].Transport;
	if (transport != null)
	{
		var localCandidates = transport.LocalCandidates;
		var remoteCandidates = transport.RemoteCandidates;
		var activeCandidatePair = transport.ActiveCandidatePair;
		var activeLocalCandidateId = activeCandidatePair.LocalCandidateId;
		var activeRemoteCandidateId = activeCandidatePair.RemoteCandidateId;

		for (var i = 0; i < localCandidates.Length; i++)
		{
			var localCandidate = localCandidates[i];
			if (localCandidate.Id == activeLocalCandidateId)
			{
				// this is the active local candidate

                // check the protocol - UDP or TCP
                var localCandidateProtocol = localCandidate.Protocol;

				if (localCandidate.IsRelayed)
				{
					// check the relay server IP
					var relayServerIPAddress = localCandidate.IPAddress;
				}
			}
		}
		for (var i = 0; i < remoteCandidates.Length; i++)
		{
			var remoteCandidate = remoteCandidates[i];
			if (remoteCandidate.Id == activeRemoteCandidateId)
			{
				// this is the active remote candidate
				if (remoteCandidate.IsRelayed)
				{
					// check the relay server IP
					var relayServerIPAddress = remoteCandidate.IPAddress;
				}
			}
		}
	}
});

FAQ

I am getting an error in Xamarin re: "dynamic registrar". How do I fix this?

If you are using Mono 5.10.x or higher there is a know issue with Xamarin iOS (or Xamarin Forms iOS) where the dynamic linker is removed at runtime. The native libraries will fail to load without the dynamic linker. A work around is to add --optimize=-remove-dynamic-registrar flag to MTouchExtraArgs in the .csproj file . Adding this flag will prevent Mono from removing the dynamic linker. Please see the Mono Dynamic Registrar Flag section of the Starting a Xamarin iOS Project discussion for an example.

This error will appear in your logs similarly to this:

[LiveSwitch] ERROR [FM] 2018/05/09-17:15:45 Error initializing local media.
ObjCRuntime.RuntimeException: Can't register the class FM.LiveSwitch.Opus.Encoder when the dynamic registrar has been linked away.