Set up live agent assistance with Conversation Intelligence and Flex

This guide shows you how to set up a Flex plugin and backend service that use Conversation Intelligence to surface real-time insights and guidance to agents during live calls. You'll clone a sample application, configure Conversation Intelligence with language operators, and deploy a Flex plugin that displays the live results directly in the agent desktop.

A screenshot of the Flex plugin displaying live agent assistance

Live agent assistance use cases

The following are some examples of how you can use Conversation Intelligence and Flex to provide real-time assistance to agents during live calls:

Triggering next best actions (both conversation prompts and system actions)
Delivering targeted responses or actions based on sentiment, for agents and supervisors
Providing live coaching, either automated or with supervisor input
Surfacing contextual data relevant to the current issue, question, or comment
Identifying sales-related opportunities (upsell, cross-sell, churn risk response, competitor mention)

Sample application architecture

In this guide, you'll learn how to build a live agent assistance solution using Twilio Flex, Conversation Intelligence, and Conversation Orchestrator. The solution provides real-time insights and guidance to agents during live calls, helping them improve customer interactions and outcomes.

There are three primary components to this solution. In this example, the components are already built for you, and you'll mainly be setting up the configuration in your Flex instance.

Backend service

The server can run on your own desktop (using ngrok), but you'd typically deploy it to your cloud platform of choice. This guide uses Node, but you could use Python or another stack of your choice. The server is responsible for receiving webhooks from the Twilio Conversations services, processing them, and providing a standardized format for use in the Flex plugin. This standard format is then piped over to the plugin via SSE (Server-sent Events), using the Call SID as an identifier.

Twilio Conversations

Use Conversation Orchestrator and Conversation Intelligence to gain insights on the active call. You can set up these services using the Twilio Console or the API.

Twilio Flex plugin

A custom plugin receives information processed by the backend service and formats for display to the human agent directly inside the Flex agent desktop. Information is displayed based on the active Task represented by a Call SID.

Prerequisites

Sign up for Twilio and buy a Twilio phone number with voice capabilities.
Install Node.js 18 or higher
Install and set up ngrok
Install Twilio CLI
Install Flex plugin CLI

Start ngrok

To start ngrok and create a tunnel to your localhost, run:

ngrok http 3001

The ngrok console displays your forwarding URL (for example, https://your-ngrok-domain.ngrok.app).

Clone the repository

In a terminal, clone the application repository and navigate to the project directory:

1git clone https://github.com/skills0/flex-agentassist-cintel.git
2cd flex-agentassist-cintel

The repository includes two main folders: /server and /plugin-cintel-rtt

Install dependencies

In each of the /server and /plugin-cintel-rtt directories, run:

npm install

Configure environment variables

Create .env files for both your server and your plugin:

/server/.env

1TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2TWILIO_AUTH_TOKEN=your_auth_token
3TWILIO_API_KEY=SKxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
4TWILIO_API_SECRET=your_api_secret
5TWILIO_FLEX_PHONE_NUMBER=+1xxxxxx1234
6SERVER_URL=https://your-ngrok-domain.ngrok.app

/plugin-cintel-rtt/.env

REACT_APP_BACKEND_URL=https://your-ngrok-domain.ngrok.app

Configure your Twilio phone number

In the Twilio Console, go to Numbers & senders > Phone numbers and click your Twilio phone number.
On the Configuration details tab, click Voice and emergency address.
Click Edit configuration details and select the region.
Under How do you want to configure this number?, select Webhook, TwiML Bin, Function, Studio Flow, Proxy Service.
Under How do you want to set up your primary method, select Use Studio Flows and select Voice IVR.
Under Call status change, enter https://your-ngrok-domain.ngrok.app/webhook/status and select HTTP POST.

Set up Conversation Intelligence

Set up Conversation Intelligence with language operators. You can use Twilio-authored language operators for common tasks like sentiment analysis and summaries, or create custom language operators tailored to your business needs.

Twilio ConsoleAPI

Create a conversation configuration with capture rules.
1. In the Twilio Console, go to Products & Services > Conversation Orchestrator > Conversation configurations.
2. Click Create a conversation configuration and follow the steps in the UI.
  1. On the Name configuration page, set Conversation configuration name and Description. Under Webhook, enter your webhook URL (for example, https://your-ngrok-domain.ngrok.app/webhook/cor-event) and select the POST HTTP method.
  2. Skip Messaging and chat traffic.
  3. On the Voice traffic page, check the Set up automatic capture box and select your Twilio phone number.
  4. On the Configure lifecycle page, select Basic and set voice calls to close On Hangup.
  5. On the Enable Conversation Memory page, select an existing memory store or create one. Check the Turn on observations and summaries box.
  6. Review the Voice transcription page.
  7. Review the Summary page and click Create conversation configuration.
Create an intelligence configuration and link it to the conversation configuration.
1. In the Twilio Console, go to Products & Services > Conversation Intelligence > Intelligence configurations.
2. Click Create Intelligence configuration and follow the steps in the UI. Make sure to attach the conversation configuration you created in the previous step.
3. Click Create rule for the intelligence configuration and follow the steps in the UI.
  - Add language operators and set parameters.
  - Apply triggers and actions.
  - Optionally, enable context.

Create a conversation configuration with capture rules and a status callback webhook so your backend server receives conversation, participant, and transcript events.

Create a conversation configuration for live agent assistance

1// Download the helper library from https://www.twilio.com/docs/node/install
2const twilio = require("twilio"); // Or, for ESM: import twilio from "twilio";
3
4// Find your Account SID at twilio.com/console
5// Provision API Keys at twilio.com/console/runtime/api-keys
6// and set the environment variables. See http://twil.io/secure
7// For local testing, you can use your Account SID and Auth token
8const accountSid = process.env.TWILIO_ACCOUNT_SID;
9const apiKey = process.env.TWILIO_API_KEY;
10const apiSecret = process.env.TWILIO_API_SECRET;
11const client = twilio(apiKey, apiSecret, { accountSid: accountSid });
12
13async function createConfiguration() {
14  const configuration = await client.conversations.v2.configurations.create({
15    displayName: "flex-conversation-configuration",
16    description: "Conversation configuration for Flex live agent assistance",
17    conversationGroupingType: "GROUP_BY_PROFILE",
18    memoryStoreId: "<MEMORY_STORE_ID>",
19    channelSettings: {
20      VOICE: {
21        captureRules: [
22          {
23            from: "YOUR_TWILIO_PHONE_NUMBER",
24            to: "*",
25          },
26          {
27            from: "*",
28            to: "YOUR_TWILIO_PHONE_NUMBER",
29          },
30        ],
31      },
32    },
33    statusCallbacks: [
34      {
35        url: "https://your-ngrok-domain.ngrok.app/webhook/cor-event",
36        method: "POST",
37      },
38    ],
39  });
40
41  console.log(configuration.statusUrl);
42}
43
44createConfiguration();

Create an intelligence configuration and link it to the conversation configuration.

Make a POST request to the Intelligence Configurations resource.

The following example creates an intelligence configuration with two rules:

Script Adherence operator (see the Script Adherence operator example for the sample script).
Sentiment and Summary operators (see the Sentiment and Summary operator example).

Create an intelligence configuration for live agent assistance

1// Download the helper library from https://www.twilio.com/docs/node/install
2const twilio = require("twilio"); // Or, for ESM: import twilio from "twilio";
3
4// Find your Account SID at twilio.com/console
5// Provision API Keys at twilio.com/console/runtime/api-keys
6// and set the environment variables. See http://twil.io/secure
7// For local testing, you can use your Account SID and Auth token
8const accountSid = process.env.TWILIO_ACCOUNT_SID;
9const apiKey = process.env.TWILIO_API_KEY;
10const apiSecret = process.env.TWILIO_API_SECRET;
11const client = twilio(apiKey, apiSecret, { accountSid: accountSid });
12
13async function createConfiguration() {
14  const configuration = await client.intelligence.v3.configurations.create({
15    displayName: "flex-intelligence-configuration",
16    description: "Intelligence configuration for Flex live agent assistance",
17    rules: [
18      {
19        operators: [
20          {
21            id: "intelligence_operator_01kf34tcyefpyb1t4m0nbd8rxg",
22            parameters: {
23              script: "",
24            },
25          },
26        ],
27        triggers: [
28          {
29            on: "COMMUNICATION",
30            parameters: {
31              count: 1,
32            },
33          },
34        ],
35        actions: [
36          {
37            type: "WEBHOOK",
38            method: "POST",
39            url: "https://your-ngrok-domain.ngrok.app/webhook/cintel-action",
40          },
41        ],
42        context: {
43          memory: {
44            enabled: true,
45          },
46        },
47      },
48      {
49        operators: [
50          {
51            id: "intelligence_operator_01kcrvw16kfa88qvgrfmr7y151",
52          },
53          {
54            id: "intelligence_operator_01kcv35pnkeysaf6z6cqtbpegn",
55          },
56        ],
57        triggers: [
58          {
59            on: "COMMUNICATION",
60            parameters: {
61              count: 5,
62            },
63          },
64        ],
65        actions: [
66          {
67            type: "WEBHOOK",
68            method: "POST",
69            url: "https://your-ngrok-domain.ngrok.app/webhook/cintel-action",
70          },
71        ],
72        context: {
73          memory: {
74            enabled: true,
75          },
76        },
77      },
78    ],
79  });
80
81  console.log(configuration.accountId);
82}
83
84createConfiguration();

To attach the intelligence configuration, make a PUT request to the Conversations Configuration resource.

Attach your intelligence configuration to a conversation configuration

1// Download the helper library from https://www.twilio.com/docs/node/install
2const twilio = require("twilio"); // Or, for ESM: import twilio from "twilio";
3
4// Find your Account SID at twilio.com/console
5// Provision API Keys at twilio.com/console/runtime/api-keys
6// and set the environment variables. See http://twil.io/secure
7// For local testing, you can use your Account SID and Auth token
8const accountSid = process.env.TWILIO_ACCOUNT_SID;
9const apiKey = process.env.TWILIO_API_KEY;
10const apiSecret = process.env.TWILIO_API_SECRET;
11const client = twilio(apiKey, apiSecret, { accountSid: accountSid });
12
13async function updateConfiguration() {
14  const configuration = await client.conversations.v2
15    .configurations("<CONVERSATION_CONFIGURATION_ID>")
16    .update({
17      displayName: "flex-conversation-configuration",
18      description: "Conversation configuration for Flex live agent assistance",
19      conversationGroupingType: "GROUP_BY_PROFILE",
20      memoryStoreId: "<MEMORY_STORE_ID>",
21      channelSettings: {
22        VOICE: {
23          captureRules: [
24            {
25              from: "YOUR_TWILIO_PHONE_NUMBER",
26              to: "*",
27            },
28            {
29              from: "*",
30              to: "YOUR_TWILIO_PHONE_NUMBER",
31            },
32          ],
33        },
34      },
35      statusCallbacks: [
36        {
37          url: "https://your-ngrok-domain.ngrok.app/webhook/cor-event",
38          method: "POST",
39        },
40      ],
41      intelligenceConfigurationIds: ["<INTELLIGENCE_CONFIGURATION_ID>"],
42    });
43
44  console.log(configuration.statusUrl);
45}
46
47updateConfiguration();

Example: Script Adherence operator

Define a rule with the Script Adherence operator to ensure agents follow a script during calls. For more information, see Script Adherence operator best practices.

This example evaluates if the agent greeted the customer, verified their identity, gathered move details, and confirmed next steps.

Operators	Parameters	Triggers	Actions
Script Adherence	`script`	Every communication (`{"on":"COMMUNICATION","parameters":{"count":1}}`)	Webhook (`{"method":"POST","type":"WEBHOOK","url":"https://your-ngrok-domain.ngrok.app/webhook/cintel-action"}`)

Script Adherence operator script

1Category: Greeting
2Action: Greet the customer with business name and agent name. 
3Required Phrase: "Thanks for contacting {company name}. This is {agent_name}."  
4Score weight:10%
5
6Category: Verification
7Goal: Confirm identity quickly
8Action: Capture customer move date and to/from cities within first 50 seconds of conversation.
9Score weight:20%
10
11Category: Move Details
12Goal: Confirm move details and answer questions
13Action: Agent collects details about the customer's move, including special items, international moves, customs, and insurance options
14Score weight:40%
15
16Category: Confirmation
17Goal: Next Steps
18Action: Agent summarizes next steps (preferred channel, video walkthrough)
19Score weight:30%

Example: Sentiment and Summary operator

Define a rule with the Sentiment and Summary operators to evaluate the sentiment of the conversation and generate a summary. This example triggers after every 5 communications.

Operators	Parameters	Triggers	Actions
Sentiment and Summary	N/A	Every 5 communications (`{"on":"COMMUNICATION","parameters":{"count":5}}`)	Webhook (`{"method":"POST","type":"WEBHOOK","url":"https://your-ngrok-domain.ngrok.app/webhook/cintel-action"}`)

Deploy the Flex plugin

Deploy the Flex plugin and test it locally.
1
cd plugin-cintel-rtt
2
twilio flex:plugins:start
Confirm you are connected to the correct profile for your target Flex account.
twilio profiles:list
To switch the profile, run:
twilio profiles:use PROFILE_ID
Deploy the UI component that agents will use.
1. Update webpack.config.js case statement to match your Twilio CLI profile name for your Flex account.
2. Copy /plugin-cintel-rtt/.env to /plugin-cintel-rtt/.env.production with your backend URL.
3. Run twilio flex:plugins:deploy using the example below.
  twilio flex:plugins:deploy --profile=<YOUR_PROFILE_NAME> --changelog "Initial" --description "Flex live agent assistance plugin example"
When the deployment completes, the Twilio CLI prints a twilio flex:plugins:release command. Run that command to release your plugin and make it active for agents.

Test your agent assistance

Start your local server.
1
cd server
2
npm run dev
Open your Flex instance and set your status to Available.
Make a call to the phone number configured in your conversation configuration.
Accept the task.

As you speak, the Agent View tab in Flex will automatically check off agent adherence steps (Greeting, Identity, and more) as the language operator detects them.

What's next

You can expand the sample application by adding a custom language operator, such as Upsell Opportunity Detection.

Regardless of which language operators you add, your plugin implementation needs to be aware of the types of language operator results it will receive. In this guide, the plugin is responsible for taking the raw output, which is normalized by the server webhook implementation, and figuring out how to display it.

Here are some examples of how you can customize the plugin to display results from your language operators:

To update the steps in your Script Adherence operator, change plugin-cintel-rtt/src/config/scriptDimension.json to match.
To show results for Summary and Sentiment operators above the transcript, update plugin-cintel-rtt/src/components/TranscriptPanel.tsx.
To change how operator results are processed before they're passed to other components as props, update plugin-cintel-rtt/src/components/OperatorResultLog.tsx.

Alternatively, you can also implement more logic in the backend server, with pre-defined message payloads for various types of results that your plugin knows how to display.