Simulcast

Overview

Simulcast is short for "simultaneous broadcast." It's used in WebRTC to improve the overall quality of a group conference.

Instead of sending just one video stream from the source to the server and then to each participant, the server also creates copies ("encodings") of the same video but encoded to different quality targets. Typically, a high-quality (full bitrate), medium-quality and low-quality (reduced bitrate) streams are encoded.

Consider a group call with Alice, Bob, Carol, and Dan:

  • Alice is sending a video using a bitrate of 2 Mbps.
  • Bob and Dan have reliable internet connections and can receive Alice's video at 2 Mbps.
  • Carol, however, has a limited internet connection and can only sustain inbound video at 1 Mbps.

In this scenario, to keep Carol from dropping the call, you must coordinate with Alice to reduce her outbound bitrate to 1 Mbps. Unfortunately for Bob and Dan, they must also see Alice at the reduced bitrate. This negative side effect is amplified and becomes more and more noticeable as you add more participants. With simulcast, you can send Carol a low-quality encoding while allowing Bob and Dan to continue receiving the high-quality encoding.

Warning

Please be aware that Client-Side Simulcast has limitations. If the video stream sender has poor network connectivity, all receivers will experience poor performance. Client-Side Simulcast depends on the sender having a strong network to handle multiple video encodings being sent. In such cases, Server-Side Simulcast is preferred to maintain high-quality video streams for receivers.

Warning

Simulcast is not supported on P2P connections. In two-way calls, the bitrate adapts to each receiver individually, with no side effects since there are no additional receivers. Only SFU connections can benefit from simulcast.

How does Simulcast Improve Stream Quality?

Simulcast is a powerful tool for providing optimal video stream quality. It works by simultaneously sending multiple feeds of different quality from server to the receiving clients, dynamically enabling and disabling them depending to the network conditions experienced by the individual receivers.

Simulcast, by itself, can't resolve all quality issues for all network conditions. However, you can use it to fix most of the quality issues you encounter. There are other mechanisms that also affect overall stream quality. An alternative to simulcast that improves overall stream quality is Adaptive Video Resolution, where you adjust the video stream's resolution, constraining it to the actual solution of the receivers' view. For example, if you send HD video at a resolution of 1280 x 720, but all the receivers display the downstream in a smaller view because the app layout dictates demoting participant views during a screen share, you can adjust the sender's resolution to not waste bandwidth.

LiveSwitch continually improves bandwidth management and adaptation policies and capabilities. Contact Support if you have a use case that you want to improve.

Use Simulcast

Simulcast is fully supported in all native platforms and partially supported in web browsers. See Feature Support for specific details of what is available.

To use simulcast, set the SimulcastMode for your VideoStream to RtpStreamId:

videoStream.SimulcastMode = SimulcastMode.RtpStreamId;

You can then use this video stream when creating your SFU or MCU upstream connection.

Video Retention

In certain scenarios, Simulcast may not suffice for users experiencing poor network conditions. For instance, a user might be on the lowest encoding layer yet still struggle with video reception due to network constraints.

To maintain optimal user experience even during challenging network conditions, Simulcast can leverage our Video Retention Policy. This feature intelligently notifies users when their connection quality necessitates switching to the lowest video encoding, and recommends disabling video to preserve communication quality.

As network issues may be transient, we also provide options to automatically re-enable video when conditions improve. The following sections detail the various configuration parameters available for tailoring Video Retention to your specific Simulcast implementation.

Video Retention Policy

You can manage this issue using the VideoRetentionPolicy on the Downstream Connection. The available options are listed below, with the default setting being DynamicRetention.

Value Description
AlwaysRetain Video will always be retained regardless of connection quality. The video will not be turned off under any circumstances.
DynamicRetention Video will be dynamically enabled or disabled based on the VideoEnabledUpdateConnectionPolicy function. While this function has a default configuration, it can be overridden with your custom function. *Further details are provided below.
NeverRetain Video will be disabled upon the first indication of failure and will remain off. When the connection detects the user's initial struggle, the video will be turned off and will not be re-enabled.

Here are some examples on how to set this policy.

downstreamConnection.VideoRetentionPolicy = VideoRetentionPolicy.DynamicRetention;

Video Enable Update Connection Policy

When using the VideoRetentionPolicy.DynamicRetention policy, the video will be dynamically re-enabled based on a default function or a custom function of your choosing. This function evaluates the number of attempts made to restore the video and returns an integer value indicating, in seconds, the delay before the next attempt. A value of -1 will stop further attempts, keeping the video off.

You can override the default behavior using the VideoEnabledUpdateConnectionPolicy function. The default function will attempt to reconnect the video up to 3 times, with an initial wait time of 30 seconds for the first attempt, increasing by 30 seconds with each subsequent attempt (i.e., 30 seconds + (number of attempts * 30 seconds)).

Below are examples of how to override this function. The custom function will take one integer parameter, representing the number of attempts made to reconnect the video, and will return an integer value indicating the wait time in seconds before the next attempt.

private int MyCustomMethod(int attempts)
{
    if(attempts >= 3)
    {
        return -1;
    }

    return 30 + (attempts * 30);
}

// Further down in your openSfuDownstreamConnection method
downstreamConnection.VideoEnabledUpdateConnectionPolicy = MyCustomMethod;

Server Notification of Video Enabled/Disabled

To ensure a seamless user experience, your application should be notified whenever a video stream is enabled or disabled. You can attach an event to the Downstream Connection by adding it to the OnVideoEnabledChange action. This action will return two parameters: the downstream connection and a boolean value indicating whether the video is enabled or disabled.

downstreamConnection.OnVideoEnabledChange += (connection, videoEnabled) =>
{
  if(videoEnabled)
  {
    // TODO: Add view back in
  }
  else
  {
    // TODO: Remove view
  }
};

Configure Simulcast

Encoding Count

Use the VideoSimulcastEncodingCount property to set the number of encodings to initialize. The default value is 2.

_LocalMedia.VideoSimulcastEncodingCount = 3;

The maximum number of encodings is 3.

Bits per Pixel

Use the VideoSimulcastBitsPerPixel property to control the bitrates used by the video encoders for each encoding. The default value is 0.05.

Warning

Please note that this needs to be set before the localMedia is initialized. Any changes after the localMedia has been initialize will not take effect.

_LocalMedia.VideoSimulcastBitsPerPixel = 0.07;

The number of bits per pixel (bpp) describes a video stream's desired quality independent of the frame size (width/height) and frame rate. Given a video source that raises frames of a particular size and at a particular rate, you can determine the target bitrate for a video encoder, such as VP8, H.264, VP9, using a simple formula:

kbps = width * height * frameRate * bpp / 1000

Bandwidth usage is optimized based on the video stream's actual characteristics. To ensure quality is consistent across devices and platforms, it is strongly recommended to use bpp for custom simulcast configurations.

For example, a full HD video camera producing 1920x1080 images at 60 frames per second (fps) using the default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

1920 * 1080 * 60 * 0.05 / 1000 = 6221kbps

On the other hand, a camera producing 640x480 images at 30 fps using the same default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

640 * 480 * 30 * 0.05 / 1000 = 461kbps

Degradation Preference

Use the VideoSimulcastDegradationPreference property to control the trade-offs that lower-quality encodings must make to achieve lower bitrates. The default value is Automatic.

_LocalMedia.VideoSimulcastDegradationPreference = VideoDegradationPreference.balanced;

The following tables describe the allowed values:

Value Description
Resolution Reduces the number of pixels in half for each secondary encoding. For example, a video source raising 1280x720 frames with three encodings results in 1280x720, 905x509, and 640x360.
FrameRate Reduces the number of frames in half for each secondary encoding. For example, a video source raising frames at 30 frames per second with three encodings results in frame rates of 30, 15, and 7.5.
Balanced Reduces both the number of pixels and the number of frames for each secondary encoding, but less than the reduction that would take place with either Resolution or FrameRate. For example, a video source raising 1280x720 frames at 30 frames per second with three encodings results in resolution and frame rate combinations of 1280x720@30fps, 1076x605@21fps, and 905x509@15fps.
Automatic Select the degradation preference from the preceding three options based on the VideoType of the video source. There are three possible video types, and each type results in a different degradation preference:
  • A VideoType of Camera translates to a degradation preference of Resolution.
  • A VideoType of Screen translates to a degradation preference of FrameRate.
  • A VideoType of Unknown translates to a degradation preference of Balanced.

Control Simulcast Behavior

You can control the simulcast behavior for your app by the following:

Enable and Disable Send Encodings

Enabling and disabling send encodings is a three-step process:

  1. Get the array of VideoEncodings.
  2. Change the Deactivated flags.
  3. Set the array of VideoEncodings.

This round-trip process ensures that the number of encodings doesn't change and the correct encodings are updated.

var videoEncodings = _LocalMedia.VideoEncodings;
videoEncodings[1].Deactivated = true;
_LocalMedia.VideoEncodings = videoEncodings;

Select an Initial Receive Encoding

Set the RemoteEncoding property of the VideoStream before opening the connection to request a specific encoding from LiveSwitch. By default, LiveSwitch attempts the highest quality encoding and downgrades it if necessary.

_Channel.OnRemoteUpstreamConnectionOpen += (remoteConnectionInfo) =>
{
    ...
    if (remoteConnectionInfo.HasVideo)
    {
        var remoteEncodings = remoteConnectionInfo.VideoStream.SendEncodings;
        if (remoteEncodings != null && remoteEncodings.Length > 0)
        {
            videoStream.RemoteEncoding = remoteEncodings[remoteEncodings.Length - 1];
        }
    }
    ...
};

Switch the Current Receive Encoding

Use the Update method after opening a connection to request a different encoding from LiveSwitch in three steps:

  1. Get the connection Config.
  2. Change the RemoteVideoEncoding property.
  3. Update the connection Config.
var config = connection.Config;
config.RemoteVideoEncoding = remoteEncodings[0];
await connection.Update(config);

To set an upper bitrate constraint based on the size of a view in your app, set the MaxReceiveBitrate of your VideoStream:

videoStream.MaxReceiveBitrate = bitrateInKbps;

LiveSwitch automatically switches the inbound stream to a more suitable encoding if a better option is available to satisfy that constraint.

Feature Support

Client Feature Matrix

Warning

Client-side simulcast is an experimental feature that has known browser issues.

Feature Web Native
Chrome Firefox Safari Edge IE (ActiveX) iOS Android Windows macOS Linux
Sending multiple encodings Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Deactivating a send encoding Yes No No Yes Yes Yes Yes Yes Yes Yes
Updating a send encoding bitrate Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Updating a send encoding frame rate No1 No1 No No Yes Yes Yes Yes Yes Yes
Updating a send encoding scale (size) Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Receiving an encoding Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Switching receive encoding Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple VP8 encodings Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple H.264 encodings Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple VP9 encodings No No No No No Yes Yes Yes Yes Yes

1 https://www.chromestatus.com/feature/6216116758118400