Turn Detection for Conversation Relay
Legal notice
Turn Detection uses artificial intelligence or machine learning technologies. By enabling or using any features or functionalities within Programmable Voice that Twilio identifies as using artificial intelligence or machine learning technology, you acknowledge and agree to certain terms. Your use of these features or functionalities is subject to the terms of the Predictive and Generative AI or ML Features Addendum.
Turn Detection isn't PCI compliant or a HIPAA Eligible Service. Don't use the turnDetection attribute in workflows that are subject to PCI or HIPAA regulations.
Learn how to use Turn Detection with Conversation Relay to detect when a caller has finished speaking. Turn Detection lets your AI voice agent wait while a caller pauses to think, and then respond promptly after the caller finishes.
To learn more about the <ConversationRelay> attributes and WebSocket messages that this guide uses, see Related reference documentation.
For other guides, see All Conversation Relay guides.
An AI voice agent needs to know when the caller has finished speaking. If the agent responds too early, it interrupts the caller. If it waits too long, the conversation feels slow.
By default, Conversation Relay relies on the speech-to-text (STT) provider's silence detection to end the caller's turn. A silence threshold can't distinguish between a caller who has finished speaking and a caller who is pausing to think or using filler words. As a result, the agent sometimes interrupts the caller.
Turn Detection replaces the silence-based decision with a semantics-aware machine learning model built by Twilio. The model evaluates the caller's audio, not the transcript. It does this in real time to determine whether the caller has completed their turn or is pausing mid-sentence. The model runs in Twilio's media stack alongside STT, so it works with all supported STT providers and speech models except Deepgram Flux. For more information about Flux, see Limitations.
When the model determines that the turn is complete, Conversation Relay sends the prompt message with last set to true, instead of waiting for an STT silence timer to expire. Turn Detection affects only when the turn ends. Turn Detection doesn't change how Conversation Relay detects interruptions of the agent's speech.
Turn Detection is most useful when callers often pause in the middle of a thought. Consider the following scenarios:
- Long values: Callers who recite confirmation codes, tracking numbers, or phone numbers tend to pause between groups of digits.
- Hesitant conversations: In technical support or claims calls, callers often stop to collect their thoughts or use filler words such as um and ah.
- Bookings and reservations: A caller might say "I'd like a table for" and pause before adding "four people on Friday night." An agent that replies after the first pause interrupts the caller.
To turn on Turn Detection, set the turnDetection attribute to true on the <ConversationRelay> noun:
1<?xml version="1.0" encoding="UTF-8"?>2<Response>3<Connect>4<ConversationRelay5url="wss://YOUR_SERVER_URL"6welcomeGreeting="Hi! How can I help you today?"7turnDetection="true"8/>9</Connect>10</Response>
Replace YOUR_SERVER_URL with the URL of your WebSocket server.
You don't need to change your WebSocket server. Your application continues to receive prompt messages and treats last: true as the end of the caller's turn. A prompt message that ends the caller's turn looks like the following:
1{2"type": "prompt",3"voicePrompt": "I would like to book a table for Friday night.",4"lang": "en-US",5"last": true6}
If your application needs more detail about the caller's turn, set the includeInPrompt attribute to turn-state. Conversation Relay then adds a turnState field to the prompt messages that it sends. To receive the start and interim turn states, also set partialPrompts to true.
1<?xml version="1.0" encoding="UTF-8"?>2<Response>3<Connect>4<ConversationRelay5url="wss://YOUR_SERVER_URL"6turnDetection="true"7includeInPrompt="turn-state"8partialPrompts="true"9/>10</Connect>11</Response>
With this configuration, the prompt message that ends the caller's turn is similar to the following:
1{2"type": "prompt",3"voicePrompt": "I would like to book a table for Friday night.",4"lang": "en-US",5"last": true,6"turnState": "end"7}
The turnState field can have one of the following values: start, interim, or end. To learn what each value means and when Conversation Relay sends it, see the prompt message with turn state reference.
Use the following attributes to balance how quickly Conversation Relay reports the end of a turn against how accurately it reports it:
| Attribute | Description | Default | Allowed values |
|---|---|---|---|
turnMinSilence | The minimum silence that Conversation Relay waits for before it reports the end of a turn. Conversation Relay doesn't end the turn until the caller has been quiet for at least this long. | 300 milliseconds | An integer from 100 to 2000 milliseconds |
speechTimeout | The time after which Conversation Relay reports the end of a turn when the model isn't certain that the caller has finished speaking. | 2000 milliseconds (3000 with Deepgram Flux) | auto, or an integer from 600 to 5000 milliseconds |
eotThreshold | How confident the model must be that the caller has finished speaking before Conversation Relay ends the turn. | 0.8 | 0.5 to 0.9 |
Set speechTimeout to at least 300 milliseconds more than turnMinSilence. turnMinSilence doesn't apply with Deepgram Flux.
With the default values, turnMinSilence="300" and speechTimeout="2000", Conversation Relay favors a faster response. It might split a single caller turn into more than one prompt report. To tune the balance, start by adjusting turnMinSilence:
- Longer turns: Increase
turnMinSilenceif callers pause often and Conversation Relay ends their turns too early. - Faster responses: Decrease
turnMinSilenceto reduce the delay before Conversation Relay reports the end of a turn.
Test eotThreshold together with turnMinSilence. A higher eotThreshold makes Conversation Relay less sensitive to low-confidence turn decisions, which results in longer turns and fewer splits.
You can change the turnDetection, includeInPrompt, turnMinSilence, speechTimeout, eotThreshold, and partialPrompts settings during a call by sending a config message over your WebSocket connection. For example, you can turn on Turn Detection for the part of the call where the caller recites a long number, and then turn it off afterward. The following message turns on Turn Detection and sets speechTimeout to 2000 milliseconds:
1{2"type": "config",3"session": {4"turnDetection": true,5"speechTimeout": 20006}7}
A change to includeInPrompt or partialPrompts takes effect immediately. A change to turnDetection, turnMinSilence, speechTimeout, or eotThreshold takes effect at the end of the caller's current turn, so send the update before the caller starts answering. If a config message has a value that is out of range, or leaves less than 300 milliseconds between turnMinSilence and speechTimeout, Conversation Relay rejects it and sends an error message over your WebSocket connection. The call continues with its current settings. For timing details and the full list of settings that you can update, see Update configuration during a call.
Turn Detection works with all supported STT providers and speech models except Deepgram Flux, which has its own turn detection model built in. If you set turnDetection to true and also set transcriptionProvider to Deepgram and speechModel to flux, Conversation Relay bypasses Twilio Turn Detection and uses the Flux turn detection model. You can still use includeInPrompt="turn-state" with Flux. To tune Flux turn detection, use the eotThreshold and speechTimeout attributes. For more information about these attributes, see the <ConversationRelay> reference.
Explore the following guides to build on what you've learned:
- Best practices for Conversation Relay: Learn how to handle caller silence, prevent false interruptions, and stream text tokens.
- Update configuration during a call: Change language, voice, and session settings over your WebSocket connection without ending the call.