Building Scalable Sync Apps: Best Practices & Use Cases

Twilio Sync is powerful software responsible for state synchronization of your applications across all different devices and desktops within milliseconds. It is widely used together with Twilio products like Flex, Voice & Video, Taskrouter, Studio, Functions, and Messaging. Some products, like Twilio Conversations, fully rely on it to send each message within a conversation to the participants.

Sync's power lies in contextual communication. You can enrich and enhance the relationship between your agents and your end users by including extra information that you collect with Sync. This contextual information, such as what customers are searching for or viewing on your webpage or in your application, will help you to make better decisions based on real customer activity and interest. The key benefit of Sync is giving each of your customers a more personalized experience through real-time data.

The Twilio Sync API allows you to store and access a wide range of data. Bear in mind that Sync is meant to glue different APIs together or build the missing functionality to satisfy your end users' requests. Rather than wait for new features to collect and display this contextual data, you can build it yourself with Sync. (However, we recommend making sure that your use case for Sync fits the best practices listed in this document. We do not recommend using Sync as your application's primary data store, for example.)

Twilio Sync's magic lies in its tremendous and limitless capabilities. Sync is indispensable for use cases like co-browsing between two parties sharing the recent on-screen activity. Live dashboarding is another common use case, where you can project real-time metrics from your application to web browsers and mobiles.

Because of Sync's powerful abilities, developers often use it with custom code to build any functionality that is missing for your specific use case. However, there is often a Twilio feature that fits this need already, such as Flex for building a fully customizable contact center or Conversations for adding chat functionality to your application.

Sync provides significant value for Flex customers and complements numerous use cases in the contact center realm. Twilio Sync is complimentary for the Flex customers, and it is included in the regular Flex user subscription. The following several use cases are very popular among numerous Flex users:

A queue dashboard is a common application of the Sync API. For example, with a Flex-based contact center, you can use Sync to power a custom dashboard and display real-time metrics that are not available in Flex Insights.

The standard process for building a queue dashboard includes the following steps:

• Build the backend service to collect the key metrics, such as users waiting online, number of calls/messages pending/completed.
• Store this information in Sync by making an API call
• Build a plug-in in the Flex UI to display this information in real-time
• Display valuable information for your business

By building a dashboard, you can improve your customer experience by ensuring that your business never misses a message or call. Customizing a Sync-based dashboard gives you more awareness into your end users' presence.

With Sync, you can build plugins to enhance your Flex instance. For example, you could use Sync to create a plugin for displaying recent calls that have been received and placed to an agent.

The basic steps to include this functionality are:

• Build the backend service to get all the Taskrouter events
• Store the recent call information in a Sync Map with data specific to an agent
• The Flex UI pulls in this information and provides access to the agent per request.

Multiple customers use Sync to store canned responses. Agents working on chat or any written channel do not write every response to common customer queries. Instead, they rely on a set of pre-written messages that they can drag & drop into a conversation with a customer to make answering questions simpler, easier, and more consistent.

• Store every canned response as a separate Sync Document. Save all the Sync Document SIDs in Sync Map or Sync List as an item.
• Link this Sync Map or Sync List SID with all the canned responses in the agents profile.
• Once the data is distributed to the agent, you can build functionality to search and filter the list of responses, or even drag them into the conversation

These are just a few of the use cases you can use Sync for. We have numerous blog posts about how useful Sync can be. Please take a look at our Twilio Blog to learn more about Sync API capabilities.

There are numerous great Sync features so that your applications are secure and "just work."

First, we'll cover tooling. The Sync API has four objects that you can work within your application:

• Sync Document
• Sync List
• Sync Map
• Sync Stream

The Sync resources listed above have differing degrees of granularity, so you'll want to choose the right one to hold your data. The following tips will help you architect your application in a scalable way from the start, rather than having to rip out a piece of functionality later.

Twilio Sync is not a database. Rather, it is a state synchronization tool. As such, please store only short-term and relevant data in Twilio Sync. It will not scale well if used as a large, primary database.

Instead, we recommend that you build the proper backend data service to collect and store your needed data, rather than using Sync as your database. This is the most scalable way to build production applications.

When you are prototyping, you may use Twilio Functions to prototype various parts of your proof-of-concept application with Sync as the backing data store. However, we do not recommend the use of Twilio Functions for large production applications due to the limitation of Sync's objects to 20 reads/second. If you use Twilio Functions and your application is growing quickly, you will not be able to keep up with the updates and will encounter throttling. However, if your application can work within the 20 read/s per object limit, Twilio Functions may be a solution for a production environment.

Sync includes several hard limits, set to protect our customers as well as our infrastructure. If you start to hit 429 errors during the traffic bursts, we highly recommend that you implement an exponential backoff retry logic as described here.

Due to these limits, your application should create and use multiple Sync objects to avoid single points of failure. For example, do not store all of your agents in a single Sync Map and update it frequently. Instead, it is better to have one Sync Map for each agent and update just dedicated items of the Map. An individual map representing a single agent would contain items for availability, support level, name, etc.

You should consider the distributed architecture of your application. Using a single global map, document, or list to store everything is an anti-pattern and results in a bottleneck with constant updates.

At its core, Twilio Sync is a set of events and subscribers. The API has infinite scaling possibilities if one-to-many principles are followed. Next, we'll look at some good and bad examples to get a better understanding of how to apply the above principles.

For example, let's look at an e-commerce business that wants to increase its revenue. Their current promotions and campaigns only bring temporary improvement, but they still don't have a good idea how their customers behave and interact with their site.

Using Twilio Sync, they could build an activity tracker to better understand what customers are seeking and why they are not completing their purchases. As an online shop owner, they want to know the most sellable and watched products, as well as when the customer is not able to finalize the purchase.

Their developers' main goal is to avoid bottlenecks and have information spread evenly over multiple Sync objects (see above). Twilio Sync fits well for the use cases when write and read operations are distributed across multiple Sync entities. In this case, they will store data for each product as a separate Sync Map.

The following is a potential architectural solution for our activity tracker, adhering to Sync building best practices:

Each product that we sell is a Sync Map. This gives the company the opportunity to update different items (name, color, product category, quantity, size, supplier, margin and etc) within each product's Sync Map without running up against limits or creating a hotspot.

Each product web page is a Sync Document. When an end user visits/views a page, the company can track that information by storing it in the corresponding Sync document.

Each step of the purchase process is also a Sync Document. It should be possible to record the last step that the customer completed during the checkout by appending this information to the Sync document.

Every customer is also a Sync Map. The company wants to know the search history and goods that a specific person was interested in. They will also want to store the customer's location in order to offer the goods in a given customer's region. This way, they avoid the poor experience where a customer sees a product on the site that is not actually available in the customer's country.

The architecture of the Twilio Sync friendly solution will look this way:

• Build the backend service to get all the search and purchase events of the goods with their Sync Map SIDs. Every time something happens (e.g., a customer searches for a product), you will add this information to the corresponding Sync map. Updating the Sync Map triggers a webhook to push this information to your backend service, where you should store it for long-term retrieval. You will want to use an optimized data store for fast writes, such as DynamoDB, Cassandra, or MongoDB.
• Store the recent goods search and checkout information in a Sync Map with data specific to the customer.
• Build UI or dashboard using Twilio Sync to visualize this information. Additionally, you can make your dashboard query-able and ensure that there is enough data to answer the questions:
  • Most searched goods
  • Most sold goods
  • Most visited goods web pages
  • Where the customers stopped during the not finished purchase

If we take the same example of an ecommerce cite, the following solution would build on Sync anti-patterns:

• Instead of using a backend service to store the data, your application relies on a Twilio Function to process the events and store them in Twilio Sync forever. (See above note about throttling.) A Twilio Function is insufficient for our example of an ecommerce site that will make many requests to Sync.
  • Better: back up your Sync data with a data store for long-term storage.
• All of the products are stored in a single Sync Map, rather than spread out over multiple Sync entities. Technically, this is possible because a Sync Map allows 1,000,000 items per object. However, this makes your one Sync Map into a single point of failure for all of your products.
  • Better: store every product in its own Sync Map. This way, only some products' Maps get updated at a time, rather than the same Map being updated every time.
• All the customers are stored in one Sync Map as well. Storing all of your customers in one Map means that the same map will be updated constantly, and you may run up against limits. Recall that there is a hard limit of 20 write/read requests per second; storing everything in one Sync Map will quickly run into this limit.
  • Instead: store each customer as their own Sync Map so only active customer maps get updated.

We've looked at how to build a scalable application with Twilio Sync as well as the most promising use cases. Now, we'll take a look at making our application secure.

There are some important facts and figures around this topic:

• Twilio Sync never logs or sends any customer data in object payloads to some telemetry system.
• Twilio Sync handles PII (Personally Identifiable Information) according to GDPR and HIPAA requirements.
• Twilio does not store Sync logs longer than 30 days.
• All Sync data is encrypted.

There are two levels of security control in the Sync SDKs:

• The Access Token issued by your application backend and secured by any method that you think is suitable. This can be the security method that you choose to authenticate your clients and provide them an appropriate Twilio token.
• The actual access control, where you decide which token gives access to which Sync object. There is a very detailed step-by-step guide on how to build the secure Sync App.

• Twilio Sync does not have batch-update functionality. Instead, you need to update the dedicated object for each change. For performance reasons, we advise against bulk updates.
• The Twilio Console does not support visualization of Twilio Sync activity. It is possible to use some alternative options, but they are not covered by Twilio standard Support procedures:
  • syncdb.io is built by Twilio's long-term partner Impekable. Just keep in mind that if you have more than three active Sync projects, you will have to pay for this tool.
  • Another (Twilion-built) visualization tool called Twilio Sync Inspector.
  • Please note: Neither tool is officially covered by Twilio Support, but they can be handy if there are a lot of Twilio Sync Instances in production
• At this time, Twilio Sync is not a regional product. At the moment, all the information goes through one region in the US.

Ready to build with Twilio Sync? Check out some of the following resources:

• The Sync Quickstarts
• State Synchronization with Sync Objects
• Sync Webhooks