Simulcast

Overview

LiveSwitch has full support for simulcast as of version 1.9.0. Partial support was available in earlier releases, starting with version 1.4.0.

Simulcast is a shortened word for "simultaneous broadcast". It is a technique used in WebRTC to improve the overall quality of a group conference.

Instead of sending just one video stream to the server, each participant can send multiple copies ("encodings") of the same video, but encoded to different quality targets. For example, a high-quality (full bitrate) and low-quality (reduced bitrate) stream would be typical.

To illustrate why this is useful, consider a 4-way group call with Alice, Bob, Carol, and Dan:

  1. Alice is sending video to LiveSwitch using a bitrate of 2mbps.
  2. Bob and Dan have good Internet connections and can receive Alice's video from LiveSwitch at 2mbps without issue.
  3. Carol, however, has a limited Internet connection and can only sustain inbound video from LiveSwitch at about 1mbps.

In this scenario, in order to keep Carol from dropping the call, LiveSwitch must coordinate with Alice to reduce her outbound bitrate to about 1mbps. This has an unfortunate side effect for Bob and Dan, who must now also see Alice at the reduced bitrate. This negative side effect is amplified and becomes more and more noticeable as you add more participants, which is where simulcast comes in.

With simulcast, LiveSwitch can send Carol a low-quality encoding, while allowing Bob and Dan to continue receiving the high-quality encoding.

Note that simulcast does not provide any benefit for 2-way calls. In a 2-way call, the bitrate simply adapts to each receiver. There are no side effects because there are no other receivers to impact.

For the same reason, simulcast is not supported for peer connections. Only SFU and MCU connections can take advantage of this feature.

The simulcast implementation in LiveSwitch is compatible with the draft IETF proposal (https://tools.ietf.org/html/draft-ietf-mmusic-sdp-simulcast-14) which uses RIDs to identify simulcast streams. Non-RID-based simulcast (using SSRCs only) is available for native platforms, but will likely be deprecated in a future release once the draft simulcast specification is finalized.

How does Simulcast Improve Stream Quality?

Simulcast is a powerful tool for providing optimal video stream quality given current network conditions. It works by simultaneously sending multiple feeds of different quality, dynamically enabling and disabling them according to who is subscribed to them, and on the receiver side, switching quickly between them as network conditions change.

So, will simulcast, by itself, solve every possible quality issue you may encounter? It depends, but we've given you the tools needed for most situations. There are other mechanisms at play which also factor into overall stream quality. For example, within the bounds of a particular encoding layer, simulcast relies on Bandwidth Estimation & Adaptation to adapt stream bandwidth as needed. Another mechanism which improves overall stream quality is Adaptive Video Resolution, where we adapt the resolution of the video stream constraining it to the actual resolution of the receivers' view. For example, if we are sending HD video at 1280 X 720, but all the receivers are displaying the downstream in a smaller view (perhaps because the application layout dictates demoting participant views during a screen share, or similar use case), then the sender’s resolution can be adjusted as needed so that bandwidth is not unnecessarily wasted.

The reality is that when considering overall stream quality simulcast is an important tool in our tool belt, but it is not the only one, and by itself cannot solve all quality issues for all network conditions. LiveSwitch 1.9.0 with simulcast enabled, has shown significant improvement in overall bandwidth handling under many scenarios, such as most broadcast use cases (which typically have less than ten senders), or most huddle use cases (where typically everyone is a sender, but most huddles have between 3 to 8 participants). For the typical broadcast or huddle use case, simulcast works very well to improve overall stream quality.

We are constantly looking to improve our bandwidth management and adaptation policies and capabilities. If you have a use-case that you believe could be improved, we want to hear about it - please contact us and let us know!

Usage

Simulcast is fully supported in all native platforms and partially supported in web browsers. See the feature matrix at the end of this page for specific details of what is and is not available.

To use simulcast, simply set the SimulcastMode for your VideoStream to RtpStreamId:

videoStream.setSimulcastMode(SimulcastMode.RtpStreamId);

That's it! Make sure this video stream is used when creating your SFU or MCU upstream connection, and you're done.

Adaptation

Simulcast is most useful when LiveSwitch can use bandwidth estimation to determine which simulcast encoding to send to which participant.

For this reason, it is strongly recommended to set your BandwidthAdaptationPolicy to Enabled in your client application (VideoStream owns the policy) and in your administration console (Channel configuration owns the policy). This is the default behaviour, so unless you have explicitly disabled it, there is no action required on your part.

Configuration

Simulcast has a sensible default configuration out of the box, but it can be tuned for custom use cases.

Some key points to note:

  • All simulcast configuration is done on your LocalMedia instance.
  • In native apps, configuration changes must be applied prior to calling Initialize.
  • In web apps, configuration changes must be applied prior to calling Start.

Encoding Count

The VideoSimulcastEncodingCount property lets you set the number of encodings to initialize. The default value is 2.

localMedia.setVideoSimulcastEncodingCount(2);

A maximum of 4 encodings will be accepted by the LiveSwitch Media Server.

Bits Per Pixel

The VideoSimulcastBitsPerPixel property lets you control the bitrates used by the video encoders for each encoding. The default value is 0.05.

localMedia.setVideoSimulcastBitsPerPixel(0.05);

The number of bits per pixel (bpp) describes the desired quality of a video stream independent of the frame size (width/height) and frame rate. Given a video source that raises frames of a particular size and a particular rate, we can determine the target bitrate for a video encoder (e.g. VP8, H.264, VP9) using a simple formula:

kbps = width * height * frameRate * bpp / 1000

Because it optimizes bandwidth usage based on the actual characteristics of the video stream and ensures that quality is consistent across devices and platforms, it is strongly recommended to use bpp for custom simulcast configurations.

For example, a full HD video camera producing 1920x1080 images at 60 frames per second (fps) using the default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

1920 * 1080 * 60 * 0.05 / 1000 = 6221kbps

On the other hand, a camera producing 640x480 images at 30 fps using the same default bpp of 0.05 would imply that the target bitrate for the video encoder should be:

640 * 480 * 30 * 0.05 / 1000 = 461kbps

Degradation Preference

The VideoSimulcastDegradationPreference property lets you control the trade-offs that lower-quality encodings must make to achieve lower bitrates. The default value is Automatic.

localMedia.setVideoSimulcastDegradationPreference(VideoDegradationPreference.Automatic);

There are four values allowed:

  1. Resolution
  2. FrameRate
  3. Balanced
  4. Automatic

Selecting Resolution will reduce the number of pixels in half for each secondary encoding. For example, a video source raising 1280x720 frames with 3 encodings will result in resolutions of 1280x720, 905x509, and 640x360.

Selecting FrameRate will reduce the number of frames in half for each secondary encoding. For example, a video source raising frames at 30 frames per second with 3 encodings will result in frame rates of 30, 15, and 7.5.

Selecting Balanced will reduce both the number of pixels and the number of frames for each secondary encoding, but less than the reduction that would take place with either Resolution or FrameRate. For example, a video source raising 1280x720 frames at 30 frames per second with 3 encodings will result in resolution and frame rate combinations of 1280x720@30fps, 1076x605@21fps, and 905x509@15fps.

Selecting Automatic will select the degradation preference from the above 3 options based on the VideoType of the video source. There are three possible video types, and each type results in a different degradation preference:

  1. A VideoType of Camera translates to a degradation preference of Resolution.
  2. A VideoType of Screen translates to a degradation preference of FrameRate.
  3. A VideoType of Unknown translates to a degradation preference of Balanced.

Control

Your application can take explicit control of simulcast behaviour at runtime. In particular, you can add code to:

  • Disable and re-enable send encodings
  • Selecting an initial receive encoding
  • Switching the current receive encoding

Enabling and Disabling Send Encodings

Enabling and disabling send encodings is a 3-step process:

  1. Get the array of VideoEncodings.
  2. Change the Deactivated flag(s).
  3. Set the array of VideoEncodings.

This round-trip process is ensures that the number of encodings does not change and that the correct encodings are updated.

var videoEncodings = localMedia.getVideoEncodings();
videoEncodings[1].setDeactivated(true);
localMedia.setVideoEncodings(videoEncodings);

Selecting an Initial Receive Encoding

The RemoteEncoding property of the VideoStream can be set before opening the connection to request a specific encoding from LiveSwitch. By default, LiveSwitch will attempt the highest quality encoding and downgrade if necessary.

channel.addOnRemoteUpstreamConnectionOpen((remoteConnectionInfo) => {
    ...
    if (remoteConnectionInfo.getHasVideo()) {
        var remoteEncodings = remoteConnectionInfo.getVideoStream().getSendEncodings();
        if (remoteEncodings != null && remoteEncodings.length > 0) {
	        videoStream.setRemoteEncoding(remoteEncodings[remoteEncodings.length - 1]);
        }
    }
    ...
});

Switching the Current Receive Encoding

The Update method can be used after opening a connection to request a different encoding from LiveSwitch. This is a 3-step process:

  1. Get the connection Config.
  2. Change the RemoteVideoEncoding property.
  3. Update the connection Config.
var config = connection.getConfig();
config.setRemoteVideoEncoding(remoteEncodings[0]);
await connection.update(config);

If you simply want to set an upper bitrate constraint, e.g. based on the size of a view in your application, then you can set the MaxReceiveBitrate of your VideoStream:

videoStream.setMaxReceiveBitrate(bitrateInKbps);

LiveSwitch will automatically switch the inbound stream to a more suitable encoding if a better option is available to satisfy that constraint.

Feature Support

Client Feature Matrix


Web Native

Chrome Firefox Safari Edge IE (ActiveX) iOS Android Windows macOS Linux
Sending multiple encodings Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Deactivating a send encoding Yes No No Yes Yes Yes Yes Yes Yes Yes
Updating a send encoding bitrate Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Updating a send encoding frame rate No1 No1 No No Yes Yes Yes Yes Yes Yes
Updating a send encoding scale (size) Yes Yes No Yes Yes Yes Yes Yes Yes Yes
Receiving an encoding Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Switching receive encoding Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple VP8 encodings Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple H.264 encodings Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes
Multiple VP9 encodings No No No No No Yes Yes Yes Yes Yes

1 https://www.chromestatus.com/feature/6216116758118400

Server Feature Matrix


Available Development Status
Simulcast on forwarded streams (SFU) Yes Shipped
Simulcast on transcoded streams (SFU) Yes Shipped
Simulcast on mixed streams (MCU) Yes Shipped
Client-driven switching to a new receive encoding Yes Shipped
Server-driven switching to a new receive encoding when the current send encoding is deactivated Yes Shipped
Server-driven switching to a new receive encoding based on bandwidth estimation Yes Shipped
Server-driven activation and deactivation of unused send encodings Yes Shipped