Large Language Models or LLMs are a form of Artificial Intelligence (AI), specifically Generative AI. An LLM is a model that was trained on a large amount of text and can generate new text. An LLM generates new text by taking an input text (typically called "prompt") and predicting what words come next. The result are models that can generate human-like content and some have even displayed reasoning capabilities to work through complex problems. This is the reason why for many, LLMs are an important step towards a world of autonomous agents and Artificial General Intelligence (AGI)
Some of the most famous LLMs are OpenAI's GPT models, Google's Gemini models, Anthropic's Claude models and open-source models such as Meta's Llama models and Mistral's Mistral & Mixtral models
Since LLMs are great at generating and processing human-like text, they are incredibly well suited for communication use cases. You can use them directly to generate content for the messages you plan to send, perform analysis on your communications by parsing through call transcripts or leveraging built-in Twilio capabilities such as Voice Intelligence's Natural Language Operators.
Increasingly LLMs are also becoming multi-modal, meaning additionally to processing and responding to text they can process audio, video, and images. This opens up further communication use cases such as using an LLM to answer questions about images that were sent via MMS or WhatsApp.
As much as LLMs are incredibly powerful, there are some real considerations to make when using LLMs or products that use LLMs. The three most common considerations are around privacy, hallucinations, and manipulation.
Once an LLM was trained, it does not magically learn new information but is essentially frozen in time. However, some AI systems might choose to collect the data sent to the model to perform additional training later to improve the accuracy of the LLM. This process is typically called "fine-tuning" and can be done through a variety of different techniques.
The important part regardless of whether an LLM gets initially trained or fine-tuned later on, it's important to understand how data is going to be used in this training process. Since LLMs use probabilities to generate their output, there is always a chance that the training data can surface in responses from the LLM.
At Twilio we use AI Nutrition Labels to make it clear to customers if and how we use data to further train the models we use, as well as which LLM we used as the base.
Since LLMs use probabilities based on the data they were trained on to generate their responses, LLMs will always try to respond with information. While this is the thing that makes LLMs intriguing in the first place, it can at times result in the LLM returning non-factual responses that are referred to as "hallucinations".
While there are mitigations to reduce the risk of hallucinations, such as leveraging retrieval-augmented generation (RAG), using fine-tuning, or using additional LLMs to "validate" the output, there is always a risk of an LLM hallucinating some content.
The nature of how LLMs are architected makes them prone to manipulation. While the types of attacks have different names such as "prompt injection", "prompt leakage" or "jailbreaks" they all aim at coercing the LLM to do something it isn't supposed to do.
This might be coercing it to reveal the prompt instructions it received to perform a task ("prompt leakage"), injecting content that modifies the behavior outlined in the original "prompt" ("prompt injection") or to break out of the safety mechanisms in the model ("jailbreaks").
Because of the risk of manipulation it's important to consider safety mechanisms not just within the LLM (by modifying your prompt) but also outside of your LLM through additional input screening, strong permissioning (if you let LLMs decide to perform tasks) and generally not considering your prompts as "private" if the remaining input is untrusted.