Media Streams - WebSocket Messages

Twilio sends the following message types to your WebSocket server during a Stream:

• Connected
• Start
• Media
• DTMF
• Stop
• Mark (bidirectional Streams only)

Twilio sends the connected event once a WebSocket connection is established. This is the first message your WebSocket server receives, and this message describes the protocol to expect in the following messages.

An example connected message is shown below.

1
{ 
2
 "event": "connected",
3
 "protocol": "Call", 
4
 "version": "1.0.0"
5
}

The start message contains metadata about the Stream and is sent immediately after the connected message. It is only sent once at the start of the Stream.

The start.customParameters object is populated with the values you provided when starting the stream. See the <Stream> TwiML doc or the Stream resource API reference doc for more info.

An example start message is shown below.

1
{
2
  "event": "start",
3
  "sequenceNumber": "1",
4
  "start": {
5
    "accountSid": "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
6
    "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
7
    "callSid": "CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
8
    "tracks": [ "inbound" ],
9
    "mediaFormat": { 
10
        "encoding": "audio/x-mulaw", 
11
        "sampleRate": 8000, 
12
        "channels": 1 },
13
    "customParameters": {
14
     "FirstName": "Jane",
15
     "LastName": "Doe",
16
     "RemoteParty": "Bob", 
17
   },
18
  },
19
  "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
20
}

This message type encapsulates the raw audio data.

An example outbound media message is shown below. The payload value is abbreviated.

1
{ 
2
 "event": "media",
3
 "sequenceNumber": "3", 
4
 "media": { 
5
   "track": "outbound", 
6
   "chunk": "1", 
7
   "timestamp": "5",
8
   "payload": "no+JhoaJjpz..."
9
 } ,
10
 "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
11
}

An example inbound media message is shown below. The payload value is abbreviated.

1

2
{ 
3
 "event": "media",
4
 "sequenceNumber": "4",
5
 "media": { 
6
   "track": "inbound", 
7
   "chunk": "2", 
8
   "timestamp": "5",
9
   "payload": "no+JhoaJjpzS..."
10
 },
11
"streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" 
12
}

Twilio sends a stop message when the Stream has stopped or the call has ended.

For unidirectional Streams, a Stream can be stopped with the <Stop> TwiML instruction or by updating the Stream resource's status to stopped.

For bidirectional Streams, the only way to stop a Stream is to end the call.

An example stop message is shown below.

1
{ 
2
 "event": "stop",
3
 "sequenceNumber": "5",
4
 "stop": {
5
    "accountSid": "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
6
    "callSid": "CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
7
  },
8
  "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" 
9
}

The dtmf message is currently only supported in bidirectional Streams.

A dtmf message is sent when someone presses a touch-tone number key in the inbound stream, typically in response to a prompt in the outbound stream.

An example dtmf message is shown below. The dtmf.digit value is 1, indicating that someone pressed the 1 key on their handset.

1

2
{ 
3
  "event": "dtmf", 
4
  "streamSid":"MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", 
5
  "sequenceNumber":"5", 
6
  "dtmf": { 
7
      "track":"inbound_track", 
8
      "digit": "1"
9
  }
10
}

Twilio sends the mark event only during bidirectional Streams.

When your server sends a media message, it should then send a mark message to Twilio. When that media message's playback is complete, Twilio sends a mark message to your server using the same mark.name as the one your server sent. Your application can use this information to keep track of which media has played on the Call.

If your server sends a clear message, Twilio empties the audio buffer and sends back mark messages matching any remaining mark messages from your server. Your application can use this information to keep track of which media messages have been cleared and will not be played.

An example mark message is shown below.

1
{ 
2
 "event": "mark",
3
 "sequenceNumber": "4",
4
 "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
5
 "mark": {
6
   "name": "my label"
7
 }
8
}

If you initiated a Stream using <Connect><Stream>, your Stream is bidirectional. This means you can send WebSocket messages back to Twilio, allowing you to pipe audio back into the Call and control the flow of the Stream.

The messages that your WebSocket server can send back to Twilio are:

• Media
• Mark
• Clear

To send media back to Twilio, you must provide a properly formatted media message.

The payload must be encoded audio/x-mulaw with a sample rate of 8000 and must be base64 encoded. The audio can be of any size.

The media messages are buffered and played in the order received. If you need to interrupt the buffered audio, send a clear message.

Below is an example media message that your WebSocket server sends back to Twilio. The media.payload is abbreviated.

1
{
2
  "event": "media",
3
  "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
4
  "media": {
5
    "payload": "a3242sa..."
6
  }
7
}

Send a mark event message after sending a media event message to be notified when the audio that you have sent has been completed. Twilio sends back a mark event with a matching name when the audio ends (or if there is no audio buffered).

Your application also receives an incoming mark event message if the buffer was cleared using the clear event message.

Below is an example mark message that your WebSocket server sends to Twilio.

1
{ 
2
 "event": "mark",
3
 "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
4
 "mark": {
5
   "name": "my label"
6
 }
7
}

Send a clear message if you want to interrupt the audio that has been sent in various media messages. This empties all buffered audio and causes any mark messages to be sent back to your WebSocket server.

Below is an example clear message that your WebSocket server sends to Twilio.

1
{ 
2
 "event": "clear",
3
 "streamSid": "MZXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
4
}