WebRTC in general is very secure and LiveSwitch follows the WebRTC spec. Here we discuss implementation specifics of LiveSwitch's security measures, as well as covering basic network related configuration.
Network Configuration and Required Ports
- STUN servers provide a WAN address discovery service and listen on port 3478 by default, but any port can be used. Firewall rules must be added to allow inbound UDP traffic on the configured port.
- TURN servers extend STUN to support relaying around restrictive firewalls. Like STUN, they listen on port 3478 by default, but any port can be used. Firewall rules must be added to allow inbound UDP and/or TCP traffic on the configured port.
- TURNS servers extend TURN by using TLS (SSL) to secure the underlying TCP connection. Since application data is already encrypted, this simply adds a layer of security on the TURN headers, but is useful to traverse firewalls that only allow TLS/SSL traffic. TURNS servers listen on port 5349 by default, but any port can be used. Port 443 is recommended since it is generally allowed by client networks. Firewall rules must be added to allow inbound TCP traffic on the configured port.
- Clients are the originator of requests and as such, they require no special firewall rules, provided outbound traffic is allowed.
- Media Server clustering requires TCP traffic on port 8445. If 8445 is unavailable the next port will be tried incrementally until one is available (i.e. 8446, 8447, 8448, 8449, etc).
- Media Server client connections require inbound UDP traffic on ports 49152-65535 on public interfaces.
- The SIP Connector requires TCP/UDP traffic on port 5060 on public interfaces.
Even the most restrictive corporate networks generally allow HTTP traffic on port 80 and TLS/SSL traffic on port 443 for HTTPS. For this reason, we recommend running a TURN server that listens on port 443 for TURNS.
All connections are secured using DTLS. Secure X.509 certificates are automatically generated for each connection using ECDSA (default, P-256 curve) or RSA (2048-bit) signing, but custom certificates and key-pairs can be provided. Certificate fingerprints (used to verify the peer during key exchange) are exchanged via signalling in the SDP blobs, so it is imperative to ensure security at the signalling layer.
- DTLS 1.2 with support for cipher suites that support perfect forward secrecy (PFS). See below for the specific cipher suites allowed by the key exchange.
- Certificates use ECDSA signing with keys generated for the P-256 curve.
- Certificates can use RSA signing with variable key length (default is 2048 bits).
- Data is encrypted using AES 128-bit encryption with 80-bit message authentication per SRTP standards.
Cipher Suites Used for DTLS
LiveSwitch connections are extremely secure once established, provided the signalling layer is also secure. Since the certificate fingerprints must be signalled in the SDP offer/answer blobs, a weakness in the signalling layer allows the possibility of a man-in-the-middle attack. For LiveSwitch, it is the Gateway that provides the signalling layer, and when installed on a server with a trusted SSL certificate, LiveSwitch's Gateway uses TLS encryption (HTTPS/WSS). The LiveSwitch Gateway also allows you to provide rate limiting, black listing, and white listing of client connections through a token-based registration model. Managing these registration tokens is an application-level security concern. We recommend an authenticated API endpoint on your own authentication server to generate and deliver registration tokens to the client securely at runtime as requested. For DDoS protection, we recommend using a third-party service like CloudFlare.
Again, all signalling connections must be secured via TLS to make security guarantees, so your LiveSwitch Gateway must be running on a server with a trusted SSL certificate, and all traffic to the Gateway must be configured to use secure connections.
TURN Traffic Concerns and Secure TURN (TURNS)
The percentage of time that WebRTC traffic is established over relay (TURN) varies greatly depending on the network environment. For example, TURN may not be required in many home networks, but in more restrictive corporate networks, where symmetric NAT is normal, requiring TURN is far more common. In general, we estimate 20%-30% of connections use TURN.
When connecting to a TURN server, clients are authenticated by username and password. Managing these credentials is an application-level security concern. We recommend an authenticated API endpoint on your own authentication server to deliver these credentials to the client securely at runtime as requested.
For added security, you can force all traffic through TURN, which guarantees that clients cannot learn information about each others' private networks (optionally including their WAN IP).
When TURN is in use, all data flows through the TURN server. Before sending data to the TURN server, clients first encrypt it using the keys exchanged via DTLS and then prepend a TURN header with destination address information. The TURN server can only read the header, as it does not have access to the keys necessary to decrypt the payload. TURNS provides an additional level of layer of security on top of this (for TCP only) by securing the TCP stream with TLS. By doing this, the entire stream is encrypted, which includes the TURN headers (and re-encrypts the payload). The TURN server is still not able to decrypt the payload, but by encrypting the entire stream, the headers are now safe from packet sniffers. This allows the traffic to pass through stateful packet inspection in routers that would otherwise block non-HTTPS traffic.
Creating an Auth Server
Creating registration tokens in your client applications is not advisable. This section will go into further detail about why this is a bad idea and will show you how to securely generate authorization tokens.
Why Not Generate Tokens Client Side?
To understand why you should not generate these tokens on the client, first consider how the LiveSwitch Gateway validates an authorization request. A authorization request is valid if the token parameters are valid values and the secret key in the token matches the secret key configured on the LiveSwitch Gateway. This means that if the client knows the secret key, that almost any request they make will be valid - there is no way for your application to stop a client from registering or joining a channel.
What this means is that the security of your application depends on keeping the secret key a secret. You can take this concept further by considering that for each token that you want to generate there are two types of data: things the client knows, and things the server knows. It's not always as black and white as that, but for our purposes here it's a useful distinction. Let's apply this concept to the parameters that are required to generate authorization tokens.
Application ID: As a unique identifier for the application, this value will come from the server. It can be hard-coded or, more commonly, the id may depend on a user's profile. Applications are associated with specific feature sets, so you do not want to allow the user to specify this as it can allow them to access features that they are not entitled to.
User ID: This value will also usually come from the server. Before a server issues an authorization token, it generally requires some form of authentication from the user. Because of this, the server knows who the user is already. You do not want to allow a client to specify this value, both because the server already knows what the value should be and because it may allow a client to impersonate another user.
Device ID: This value will generally come from the client. The device id is a unique identifier associated with a particular device. This is used in the case where a user might be connected on multiple devices, ie: through a web browser and a phone.
Client ID: This value will always come from the client. The client id is unique identifier associated with a specific LiveSwitch instance, so it must be provided by the client.
Roles: Roles will generally come from a combination of both the client or the server. In the case where an application role is associated with a particular application privilege (ie: moderation abilities), then the role will be assigned based by the server based on the user's profile. In the case where a role is not associated with any privileged action, (ie: a presenter in a conference), then client should request this role. Of course, the server should validate all client roles based on their user profile.
Channel Claims: Channel claims usually come from the client, as they represent a claim that a client makes, ie: that they have the right to join the specified channel. The server may also add other channels as required, if there are default channels or specific meta data channels associated with an application.
Shared Secret: This arbitrary string will always come from the server, as mentioned above.
Now that you have a better understanding of the concept of client information and server information, the next step is to set up your server.
Setting Things Up
To set up your auth server, you will need to set up a few things first. Generally, you will already have some sort of server to handle application-specific data such as user profiles. This is a good starting point for your auth server, as the application server will already have access to your user data, which you will need to authorize your client requests.
Generally, you will need to set up at least two additional endpoints on your server - one for generating registration tokens and one for generating channel tokens. This guide will assume a rest-like API, and will refer to these endpoints as "/token/register" and "/token/join"; meaning to register with the server, and to join a channel. In the following sections, some examples of server endpoints are provided using ASP .NET Core for C# and Spring 4 for Java.
Handling Registration Tokens
One of the main roles of an auth server is to authenticate clients and to return registration tokens so that these clients can connect to the LiveSwitch service. Registration tokens are typically generated from a "/token/register" endpoint on a REST-ful web service. In a normal registration request, the application id, user id and shared secret are provided by the server. The only remaining required parameters are the device id and the client id (because roles and channel claims are optional for a registration).
The following code snippet shows platform-specific examples of retrieving device and client ids and then making an http request to the server for a registration token. The device id and client id are provided via query parameters in a POST request.
You will have to configure your application server to receive this request. The application server should then respond by generating a client registration token. You can generate this token using the values the client provides, and values available to the server. Pass all these values to the
GenerateClientRegisterToken static method of the
FM.LiveSwitch.Token class to return a token. This is demonstrated below.
Note that the details of how these controllers' shared secret property is provided is left up to you. A common way of doing this is through your dependency injection container. Another way of accessing it is to query your global application settings. Whatever way you select, it should not be accessible to any clients.
Handling Channel Tokens
The other main role of an auth server is to return channel tokens so that users who have already registered can join additional channels without needing to re-register. Channel tokens are usually generated from a "/token/join" endpoint on a REST-ful web service.
For a join token, the client must again provide its deviceId and clientId. It must also provide the id of a channel it wishes to join. The code samples below show how to make a REST-ful request with this information embedded in the query string.
Again, you will have to configure a "/token/join" endpoint to receive this request. The endpoint should respond with a join token that allows the client to join their requested channel. You generate this token using the
GenerateClientJoinToken static method of the
FM.LiveSwitch.Token class. Note that you must wrap the channel id provided by the client in an instance of
Upgrading your Server
LiveSwitch is a core product at Frozen Mountain Software and we are constantly making improvements to it. Because of this, it's important to keep your LiveSwitch server upgraded, so that you can take advantage of the latest features. We make this process as easy as possible. To upgrade to the latest version quickly and painlessly, simply download the latest LiveSwitch SDK version and run the installer. Your pre-existing settings will be saved, so you will not have to re-configure your servers.
Which Component Do I Upgrade First?
If your gateway and media server are on the same host, then you don't need to worry about this. The installer will update them both at the same time. If your services are on different hosts, then it is only a bit more complicated. The golden rule is to always upgrade your gateway service first. The gateway service is backwards compatible with older versions of LiveSwitch. By upgrading the gateway first, you ensure that there are no interruptions for any clients that are using older versions of LiveSwitch. If you upgrade your client applications or media servers before your gateway, then you run the risk of the gateway rejecting your connections because it has not been upgraded yet. The gateway service will never accept connections from newer versions of LiveSwitch clients, because it cannot guarantee that it supports the same feature set expected by the client.
How Much Downtime Can I Expect?
If all of your services are on one host, the downtime will be minimal. The installer will terminate each service, and restart it immediately after the upgrade. If your services are on separate hosts, the upgrade should be painless as well, provided you follow the upgrade order described above. There will be a brief interruption as each service is restarted, and the service will be available again afterwards. The next question addresses scenarios where seamless upgrading is required.
How do I Gracefully Terminate Services In Use?
To ensure minimal downtime for your users and also that your upgrade process does not interrupt their use of your service, you must have at least two gateway servers and at least two media servers. With two media servers, you can upgrade each one individually, ensuring that there are no service interruptions. Through the gateway administration panel you can tell a media server to "deactivate" itself. This is a slow shutdown - the media server will not accept any new connections during this time. Once you have determined that it is safe to do so (no connections remaining) it is up to your Dev-ops team to shut down the media server. By design media servers will not shut down automatically. Once the media server has been shut down you can upgrade it without interrupting any users' session. You can do this for each media server in turn, until all are upgraded.
For a seamless upgrade of gateway services, you must ensure that you have at least two gateway services and that they are both are configured to use a redis backend, rather than the default in-memory backend. The redis backend ensures that existing clients' connection information is propagated to all gateway servers. The extra gateway service ensures that when a gateway service goes down, your clients will seamlessly reconnect to the next available gateway service.
Are There Any Plans to Improve the Upgrade Process
Yes! We love automating things, and we realize that having to manually "drain" each media server before an upgrade can take unnecessary time out of a busy developer's day. In the future, we plan to make this the default upgrade behavior.
The LiveSwitch Linux distribution does not have an installation script, so upgrading requires slightly more manual intervention. Follow the same basic guidelines for a Windows upgrade. However, instead of running the installer, stop each service using
systemctl, copy the new files to the LiveSwitch service's directory, and then restart the service.
In the future we will provide both rpm and deb packages for easy installation.
Regionality is a feature that improves clustering performance by allowing each client to select their preferred region. A region is an identifier that represents a specific geographic area. This allows each client to select a geographic area close to them, which ensures fast connection times and high throughput.
What regions are available?
The region identifiers are specific to your application. If your application has many users that are concentrated in a few areas, you will likely have regions with small geographic areas, such as "chicago" or "new-york". If, on the other hand, you have a global user base, you will instead want to use regions that represent larger geographic areas, such as "north-america" or "europe".
Where do I set the list of regions?
There is no canonical set of regions. Each media server can select its own region, and the set of available regions is taken from each media server that is connected to the current cluster.
How does region assignment work?
When a client requests a specific region, the gateway service will attempt to assign it to a media server in the same region. If there are no available media servers in the requested region, or all of the media servers in the requested region are over capacity, the gateway will ignore the region and assign the client to an available media server on a round robin basis.
What this means is that a client's selected region is only a preference and that clients requesting a specific region can potentially be assigned to any media server in the global pool if no media servers for their region are available.
Setting up Regions
There are two steps to setting up regions. First, you must assign a region to each media server that will use the regionality feature. Second, you must add an extra parameter to your logic that generates the client registration token.
Assigning a region to a media server
To assign a region to a media server, update the Server Specific Configuration.
Updating the client registration token
Once your pool of media servers have been assigned regions, you must update the client registration logic so that each client requests a specific region. You can review the client documentation for your specific platform to see how to generate tokens and join channels, but to request a specific region you simply specify the optional
regionId parameter when you create the client registration token. This parameter comes after the
sharedSecret parameter, and indicates the region that the client would like to be assigned to. An example of this is below:
You should never generate the token client side in production. We're doing so for demo purposes ONLY. Refer to the section on Creating an Auth Server for more information.
TURN in the Media Server
LiveSwitch now integrates TURN server functionality directly into your Media Server. No longer do you have to maintain separate server infrastructure for your TURN and secure TURN (TURNS) servers. Now, you simply configure TURN bindings in your LiveSwitch Configuration Console, and the rest takes care of itself.
As an added bonus, this improves the overall efficiency of relay connections, as the latency between the TURN server and the Media Server disappears, and thanks to Regionality, you can easily ensure that your clients are connecting to Media Servers that are nearby.
Configuring TURN/TURNS Bindings
TURN and TURNS Bindings are configured in the Advanced Configuration section of the LiveSwitch Configuration Console. You will also need to edit the Server Specific Configuration on each Media Server.
The Public Hostname is necessary for TURNS, because all traffic to the server must include this host name, which is then compared to the host name that is part of your certificate. Only when they match are TURNS connections possible.
This presents an interesting Operations problem: as you spin up Media Servers to accommodate increasing load, you must also have a mechanism that creates the host name for the embedded TURNS server this new Media Server will use. Your Ops process will have to create the necessary DNS record, and also supply the new host name as part of the TURNS configuration of the new Media Server. Conversely, as you tear down Media Servers you can clean up the DNS records for the TURNS domain names that are no longer required.
How Does it all Work?
No doubt you are well aware that it is typically clients that supply ICE servers, and provide these to a connection, which in turn gathers ICE candidates, etc. So, how is this all working? Obviously the Media Server can make use of its own embedded TURN functionality, but how are clients gaining access when they have not yet opened a connection to the Media Server?
The simple answer is the Gateway. When your Media Server registers with the Gateway, it lets the Gateway know about its TURN configuration. Clients in turn get their ICE server credentials from the Gateway before opening a connection provided (a) you have not already supplied ICE servers to the pending connection, and (b) the connection has the
DisableAutomaticIceServers property set to
Embedded TURN and P2P Connections
Ok, you’re thinking, this all makes a lot of sense for SFU/MCU connections, but what about my P2P connections, they do not connect with the Media Server. How are P2P connections getting access to the ICE servers they require?
It’s true that P2P connections do not even touch the Media Server, but they do register with the Gateway, and just like SFU/MCU connections, will query the Gateway for ICE servers before opening a connection. A very nice side effect here is that even P2P connections now benefit from your Regionality configuration. If you have configured Regionality properly, then clients will in turn use the region appropriate Media Server for relay (where relay is necessary), which can in turn mitigate latency.
High Availability Architecture
v1 REST API
The REST API is hosted by the LiveSwitch gateway. You can access the REST API provided you have IP-level access to the Admin endpoint (/admin by default). For example:
Server Specific Configuration
While the majority of your configuration is done using the LiveSwitch Configuration Console, there are a few server specific settings that need to be set in a configuration file in the server directory. Each service has their own configuration file in their install location:
- Gateway: FM.LiveSwitch.Gateway.Service.config.json
- Media Server: FM.LiveSwitch.MediaServer.Service.config.json
- SIP Connector: FM.LiveSwitch.Connector.Sip.Service.config.json
The following settings can be set in all the configuration files.
Redis Connection String - The connection string to the Redis server that contains the configuration. The format of the connection string is defined here. This string is mandatory and all services must connect to the same Redis server or they will not work correctly.
Region: Specifies the region of the server for Regionality.
Media Server Configuration
The Media Server configuration includes extra settings required for the Embedded TURN Server.
External IP Address: The local IP Address(es) of the Media Server.
Public IP Address: The IP Address of the Media Server, as seen by remote clients. If this value is not set then the External STUN Servers are used to retrieve it.
Public Hostname: The hostname of the Media Server. This value is mandatory if you have configured TURNS Bindings.
How Do I Reset the Configuration Console Password?
You can reset the password to your LiveSwitch Console using the command line LiveSwitch CLI Tool that is installed with your Gateway(s).
Do I need a STUN/TURN server with LiveSwitch?
In general, LiveSwitch does not require STUN or TURN for SFU/MCU connections. LiveSwitch does require STUN/TURN for guaranteed peer connections.
There are several use cases for STUN and/or TURN:
- STUN/TURN: When both endpoints in a streaming connection reside behind a NAT/router. In this case, STUN and/or TURN assists with NAT traversal.
- TURNS: When an ISP actively blocks real-time media using packet inspection. In this case, using the TLS-encrypted TURN transport bypasses prying eyes.
- STUN: When you want to know your own public IP address. In this case, the STUN server is used to simply echo back the public IP address of the client that made the request.
Use case 1 is why LiveSwitch requires STUN/TURN for peer connections. It also explains why STUN/TURN is generally not required for SFU/MCU connections. Since a publicly-accessible LiveSwitch media server is one of the endpoints in SFU/MCU connections, NAT traversal is not required. Note that if all endpoints are on the same local subnet, then STUN/TURN is generally not required.
Use case 2 is why we say that "in general" LiveSwitch doesn't require STUN or TURN for SFU/MCU connections. An Internet service provider can throw a wrench into the middle of things by actively blocking media packets. This happens more often on networks where an anti-competitive practices are in place to ensure that customers use a pre-selected media provider (e.g. on hotel networks). Adding a TURNS server (and configuring clients to use it in their ICE server array), ensures a safe fallback.
Use case 3 only applies to the media server, and is entirely optional. When deploying a LiveSwitch media server to the public Internet, there are two possible options for IP addressing:
- The server can bind directly to a private IP address, which is 1:1 port-mapped to a public IP address.
- The server can bind directly to a public IP address.
The LiveSwitch media server needs to know its public IP address so it can negotiate streaming routes with clients. If you are using option 1, which is typical of a cloud compute hosting provider, then the media server needs to either be configured manually with its public IP address, or given the address of an external STUN server that can be used to auto-discover it. If you are using option 2, then the LiveSwitch media server can read the public IP address directly from the operating system - no special configuration is required.
- Clients require STUN/TURN(S) when creating peer connections (unless all peers belong to the same local subnet).
- Clients may require TURN(S) when creating SFU/MCU connections if their Internet service provider actively blocks real-time media.
- Media servers can optionally use STUN to auto-discover its public IP address, if the server is configured to bind to a private IP address and manual configuration of the public IP address is not desired.
We want to run our STUN/TURN server behind a proxy. How do we accomplish this?
You don't! STUN/TURN cannot go behind a proxy. The entire purpose of TURN is to inspect the IP traffic directly. TURN requires access to the unmodified original IP headers to forward traffic between peers. This is not possible where a proxy is involved.
How many server instances will I need to support my user base?
Estimating exactly the number of servers you need depends very heavily on expected concurrency. It doesn't matter how many users you have - it matters how many users you expect to have active at the same time.
As a general best practice, we estimate concurrency using the 100:10:1 rule, which states that for every 1,000 named users, 100 are quite active, and about 10 are concurrent at any given point in time. Using 32,000 as a starting point, that means we need to plan for 3,200 active users and 320 concurrent at any moment in time.
In general you can expect one Media Server, on the equivalent of an AWS-EC2 c4.large (2xCPU, 4GB RAM) instance, to handle ~100 concurrent connections in SFU mode with a standard 480p stream. So, to handle the load described here you would require at least four Media Servers in a clustered environment.
A single Gateway can handle a lot of traffic, far more than our example of 320 concurrent connections. We run our demo environment Gateway on the equivalent of an AWS-EC2 c5.xlarge (4xCPU, 8GB RAM) and it can handle thousands of concurrent clients. That said, we recommend running two Gateways behind a standard load balancer for redundancy purposes.
For MCU and SFU connections relay is generally not required. See the FAQ question Do I Need a STUN/TURN Server with LiveSwitch? for an in depth discussion on the reasons for this. That said, if you do use P2P connections then you can expect that some portion of these will be over relay. We estimate that about 20-30% of P2P connections will require TURN, so with the example numbers discussed here we need to plan for 96 concurrent TURN users. For this example load the recommended server is also the equivalent of an AWS-EC2 c5.xlarge. TURN is very demanding on bandwidth. A typical standard-definition audio/video stream is about 1mbps, so with 96 relayed connections, you can get away with a single server that has 100Mbps of available upstream/downstream bandwidth. For redundancy, you will need 2 servers. No load balancer is required - round-robin DNS to spread the load across the two is generally considered to be best practice.
I've set up a Gateway. Now how do I tell if it's running properly?
Your Gateway must be available over HTTPS. The simplest, debugging 101, test to see if your Gateway is available is to browse to it in a browser. Let's say you have a Gateway served from mydomain.liveswitch.com, if you browse to https://mydomain.liveswitch.com/sync then you should see a page that says "LiveSwitch", the version number, and nothing else. If you do not, then your Gateway is not being served properly. If you do see the message but still cannot get a connection, then something else is wrong and you should contact email@example.com.
You can also start up your Gateway in interactive mode from the command line on the server where the Gateway service is installed. If you fire it up, it'll popup a console window that shows the Media Server registration(s) that come in along with the client requests from client app(s). This is really useful for debugging. To do so you may need to stop the Gateway Windows service first since it's installed as a Windows service. Then, go to Program Files/Frozen Mountain Software/LiveSwitch/Gateway/FM.LiveSwitch.Gateway.Service.exe.
I've set up a Media Server. Now how do I tell if it's running properly?
The Media Server can be started up from the command line in interactive mode on the server where the Media Server service is installed. If you fire it up, it'll popup a console window that shows it's registering, then registered, (along with other info) then reporting thread started. To do so you may need to stop the Media Server Windows service first since it's installed as a Windows service. Then, run the following - Program Files/Frozen Mountain Software/LiveSwitch/Media Server/MediaServer.exe