Managing Codecs

The codecs supported by Twilio's client SDKs are platform dependent. In the JavaScript SDK it's up to the browser vendors to provide codec implementations, while on mobile SDKs it depends on the device's capabilities.

Video Codecs Supported by Twilio's Programmable Video SDKs.

Check the browser support per platform table.

Note: Safari introduced webrtc support in version 11. The rest of the document will not refer to any earlier versions as they are not supported

[1]: VP9 support was added in Safari 15.

[2]: Twilio Video for iOS relies on hardware support for H.264. Hardware encode/decode is supported on all iOS devices.

[3]: Twilio Video for Android relies on hardware support for H.264, which depends on devices' capabilities. Reference the following snippets to check if the device supports H.264

Video Android 2.x - 5.x

1

2
boolean isH264Supported = MediaCodecVideoDecoder.isH264HwSupported() &&
3
            MediaCodecVideoEncoder.isH264HwSupported();

Video Android 6.x+

1

2
HardwareVideoEncoderFactory hardwareVideoEncoderFactory =
3
        new HardwareVideoEncoderFactory(null, true, true);
4
HardwareVideoDecoderFactory hardwareVideoDecoderFactory =
5
        new HardwareVideoDecoderFactory(null);
6

7
boolean h264EncoderSupported = false;
8
for (VideoCodecInfo videoCodecInfo : hardwareVideoEncoderFactory.getSupportedCodecs()) {
9
    if (videoCodecInfo.name.equalsIgnoreCase("h264")) {
10
        h264EncoderSupported = true;
11
        break;
12
    }
13
}
14
boolean h264DecoderSupported = false;
15
for (VideoCodecInfo videoCodecInfo : hardwareVideoDecoderFactory.getSupportedCodecs()) {
16
    if (videoCodecInfo.name.equalsIgnoreCase("h264")) {
17
        h264DecoderSupported = true;
18
        break;
19
    }
20
}
21

22
boolean isH264Supported =  h264EncoderSupported && h264DecoderSupported;
23

Furthermore, not all browsers support H.264. Developers can evaluate H.264 browser support usig the following code snippet

1
let isH264Supported;
2

3
/**
4
 * Test support for H264 codec.
5
 * @returns {Promise<boolean>} true if supported, false if not
6
 */
7
function testH264Support() {
8
  if (typeof isH264Supported === 'boolean') {
9
    return Promise.resolve(isH264Supported);
10
  }
11
  if (typeof RTCRtpSender !== undefined
12
    && typeof RTCRtpSender.getCapabilities === 'function') {
13
    isH264Supported = !!RTCRtpSender.getCapabilities('video').codecs.find(({ mimeType }) => mimeType === 'video/H264');
14
    return Promise.resolve(isH264Supported);
15
  }
16
  if (typeof RTCPeerConnection === 'undefined') {
17
    isH264Supported = false;
18
    return Promise.resolve(isH264Supported);
19
  }
20

21
  let offerOptions = {};
22
  const pc = new RTCPeerConnection();
23
  try {
24
    pc.addTransceiver('video');
25
  } catch (e) {
26
    offerOptions.offerToReceiveVideo = true;
27
  }
28

29
  return pc.createOffer(offerOptions).then(offer => {
30
    isH264Supported = /^a=rtpmap:.+ H264/m.test(offer.sdp);
31
    pc.close();
32
    return isH264Supported;
33
  });
34
}
35

36
// Now we can call testH264Support to check if H.264 is supported
37
testH264Support().then(isSupported => {
38
  console.log(`This browser ${isSupported
39
    ? 'supports' : 'does not support'} H264 codec`);
40
});

Audio Codecs Supported by Twilio's Programmable Video SDKs

Before establishing a multimedia communication, the involved parties need to agree on the codecs to be used. Twilio manages this through an automatic codec negotiation that imposes some interoperability restrictions.

In a Video Room, an SFU (Selective Forwarding Unit) mediates among clients. If client A wants to send media to clients B and C, then A sends only one media stream to the SFU that, in turn, forwards it to B and C. Due to this, the codec negotiation takes place based on the following principles:

• For inbound tracks (i.e. tracks getting into the SFU), the SFU tries to accept the codec preferred by the client as long as this codec is supported by the SFU.
• For outbound tracks (i.e. tracks getting out the SFU), the tracks are only offered in the codec in which they are being received.

These have interoperability implications. First because the only codecs allowed are the ones supported by the SFU. The following tables summarize them:

Video Codecs supported in Video Room SFU infrastructure

Audio Codecs supported in Video Room SFU infrastructure

Second, because the codecs negotiated at some point in a call may limit the interoperability with clients joining later. As a result, a set of clients supporting a common codec does not guarantee that they will be actually able to communicate. To illustrate this, imagine an example scenario in which two participants A and B, using Chrome and Firefox respectively, start communicating using VP8. If later a Safari < 12.1 browser joins (as C) it will not be able to receive A's and B's video tracks because Safari < 12.1 only supports H.264. This is unfortunate because both A and B could communicate H.264 but when they connected to the SFU negotiated their default that is VP8. This example scenario is represented on the following figure:

Expand image

For guaranteeing video interoperability with Safari < 12.1 we can force all the clients of a Room to publish their videos using H.264. This can be achieved using two alternative mechanisms:

• When the Room is created using the Rooms REST API, you can set the following parameter: VideoCodecs=H264.

• You can use the Room Settings menu located at your Twilio Video Console and set H.264 as the Room's video codec, as shown on the following figure:

  Expand image

When doing so, the SFU rejects to accept tracks published in VP8 and forces all the clients to use H.264, which guarantees interoperability with Safari < 12.1. The resulting Room topology in this case is the one shown below:

Expand image

For further details about using Safari < 12.1 in Rooms see the Working with Safari < 12.1 Guide.

For optimizing applications in media quality or battery life sometimes it is interesting to override codecs defaults at the client side. Due to this, Twilio's APIs have introduced a Codec Preferences capability that is available at the SDKs listed in the table below:

The Codec Preferences based on the following principles:

• It allows selecting both the preferred audio codecs and the preferred video codecs in which media tracks are published by a client SDK.
• Selected codecs are provided as an ordered list of codec specifications, so that codecs specified first have higher preference.
• Only codecs supported at a given SDK can be specified as preferred.
• Codec Preferences are set when a participant connects to a room as part of the ConnectOptions.
• The fact of a participant selecting a preferred codec does not guarantee the SDK to chose such codec. The preference just means that the codec shall have higher priority among the list of possible codecs to use. The actual chosen codec may depend on other aspects such as the codecs supported by the other parties of the communication.

The following code snippets illustrate how a Participant can connect to a room preferring iSAC as audio codec and H.264 as video codec.

Selecting preferred codecs in JavaScript SDK (required v1.3+)

1
// Web Javascript
2
const room = await connect(token, {
3
  preferredAudioCodecs: ['isac'],
4
  preferredVideoCodecs: ['H264']
5
});

Selecting preferred codecs in Android SDK (required v2.0+)

Note that checking for H.264 hardware support changed from v5.x to v6.x. See this section to learn how to use the correct syntax.

1
// Android Java
2

3
// Prefer H264 if it is hardware available for encoding and decoding
4
VideoCodec videoCodec = isH264Supported ? (new H264Codec()) : (new Vp8Codec());
5
ConnectOptions connectOptions = new ConnectOptions.Builder(token)
6
    .preferAudioCodecs(Collections.singletonList(new IsacCodec()))
7
    .preferVideoCodecs(Collections.singletonList(videoCodec)
8
    .build();
9

10
Room room = Video.connect(context, connectOptions, listener);

Selecting preferred codecs in iOS SDK (required v2.0+)

1
// iOS Swift
2
let options = TVIConnectOptions.init(token: accessToken block: {(builder: TVIConnectOptionsBuilder) -> Void in
3
    builder.preferredAudioCodecs = [ TVIIsacCodec() ]
4
    builder.preferredVideoCodecs = [ TVIH264Codec() ]
5
}
6

7
var room = TwilioVideo.connect(with: options delegate: self)

1
// iOS Objective-C
2
TVIConnectOptions *options = [TVIConnectOptions optionsWithToken:self.accessToken
3
                                block:^(TVIConnectOptionsBuilder * _Nonnull builder) {
4
    builder.preferredAudioCodecs = @[ [TVIIsacCodec new] ];
5
    builder.preferredVideoCodecs = @[ [TVIH264Codec new] ];
6
}];
7

8
TVIRoom *room = [TwilioVideo connectWithOptions:options delegate:self];

For further information on how Codec Preferences work, please check the corresponding SDK reference documentation regarding the connect primitive and its ConnectOptions parameter.

In Twilio Programmable Video, we say that a room is using Multi-codec capabilities when different participants send their video (or audio) using different codecs. Multi-codecs may be useful for optimizing the battery life of your applications as many vendors provide hardware acceleration support for specific codec suites. For example, all iOS devices have H.264 hardware acceleration while many modern Android devices provide it for both H.264 and VP8. When different devices have different hardware support, it may be interesting to send media using one codec (i.e. the one supported by the local device hardware) while receiving in another (i.e. the one hardware-supported by the remote parties).

This can be achieved using Codec Preferences. For example, imagine three clients A, B and C being respectively Chrome, Firefox and an iOS smartphone. By default (see sections above) these clients negotiate VP8 as the video codec. However, we may prefer to use H.264 in C to optimize the battery life of the iOS smartphone. In this case, we just need to set H.264 as C's preferred codec. The following code snippets illustrate how to it:

Participant A - Codec Preferences

1
// We don't really need this: VP8 is Chrome's default preferred video codec
2
const room = await connect(token, {
3
    preferredVideoCodecs: ['VP8']
4
});

Participant B - Codec Preferences

1
// We don't really need this: VP8 is Firefox's default preferred video codec
2
const room = await connect(token, {
3
    preferredVideoCodecs: ['VP8']
4
});

Participant C - Codec Preferences

1
// iOS Swift
2
let options = TVIConnectOptions.init(token: accessToken block: {(builder: TVIConnectOptionsBuilder) -> Void in
3
      builder.preferredVideoCodecs = [ TVIH264Codec() ]
4
}
5
var room = TwilioVideo.connect(with: options delegate: self)

If A, B, and C connect to a Room, the topology is the following:

Expand image

• Multi-codec support for video is currently limited to H.264 and VP8.
• Multi-codec support for audio is currently limited to Opus and PCMU.
• Support for H.264 video is currently limited to the Constrained Baseline profile at level 3.1. This means that the maximum resolution of a video track is 1280x720 at 30 fps.
• The use of multi-codecs in a Room with recording activated causes recordings to be also multi-codec. This means that a recording keeps the codec of the track originating it.