Sign in to continue learning
Context Engineering
7h55m
If you've used AI agents out in the office, or the lab, or the classroom, then you've likely bumped into issues. Maybe your agent is taking longer to generate a response, or you're hitting expensive limits on usage, or you need to interrupt your session repeatedly to provide reminders to your agent, as though it keeps missing key details about tasks. Commonly, these issues relate to too little information, and sometimes too much information. Namely, too many tokens in the context window. In the lab, we made bar charts showing token usage of the model, broken down by message types across iterations of the tool calling loop. For a prompt like, what are the five most common topics among customer service messages, the context window piled up across iterations with tool outputs. That makes sense, because tools build up context by searching and retrieving data for the model. But before you know it, too little context becomes too much context, sometimes leading to unexpected responses, well before limits on context length. Context length is the largest number of tokens the model can hold in its context window. Older models had context length around 8,000 or 16,000 tokens. But these days, we have 200,000 tokens or more, with several models reaching one million, while that might seem like a lot of data. One million tokens is usually less than five megabytes of data. So, suppose every turn grows the context by 10%. That means context doubles every seven turns. If you start with 1,000 tokens, which is about a page and a half of single space text, then you've reached the one million token limit in about 70 turns. Seventy turns of a conversation would be a lot for you or me, but not for an agent. For agents, the context window can fill up quickly, especially when it's working on tasks with multiple steps, each taking seconds or minutes or even hours. Actually, a study measuring the amount of time taken by agents to complete tasks shows a trend that's up and to the right, doubling about every seven months, from a few seconds to a few hours. So, as agents get more capable, working on harder tasks over longer periods, prompts, tools, memory, and traces can pile up in the context. And some of that context, like system prompts, is part of the harness. When you use popular harnesses from providers like Anthropic, OpenAI, and Google, there's a system prompt containing thousands of tokens, much of it describing tools and composition of tools in particular workflows. The context window can fill up with tokens all the way to the context length, without any errors thrown by the model. However, you're going to incur more cost for a less fast and less attentive model, sometimes after just 20% utilization of the context window. So, for a context length of one million tokens, that's 200,000 tokens, the model slows down because it has more tokens to process, and it struggles to pay attention to the details across all those tokens. That gap between the context length and the point where you encounter some degradation in performance is sometimes called the effective context length. And while you've probably encountered it yourself, agents that seem to get sluggish and lazy, it's not just anecdotal. Chroma showed empirically, in a seminal research report called Context Rot, which we've linked in the reading items for Module 1, that the gap between context length and effective context length can be large, even for popular models from leading AI labs. So, let's understand Context Rot by discussing cost, speed, and attention in some more detail. These days, you might have one or more subscriptions to different AI services, each with its own costs. But when you build agents, you use models as a service through the provider's API. And for that, you pay by input token to the model, and output token from the model. Remember, a token is a word, or fragment of a word, that gets assigned a number, which the model can associate with one of about 10,000 to 100,000 distinct tokens in its vocabulary. Depending on the model, model providers set a price per input token, and another, often higher price, per output token. Note that you incur a charge for both input tokens and output tokens, because both appear in the context window during inference. That's because the model writes one token at a time, and to generate the next token during inference, the model must re-read every token before it. That's the streaming you see in ChatGPT. It's not an interface effect. The model generates left to right until it hits a stop token. So, more context means more tokens to re-read, leading to slower output. Based on my experiences with older models, at around 1,000 tokens, a model generates about 10,000 tokens a second. So, 10 Ks. But at 128,000 tokens, that drops to about 0.13 Ks. There are different benchmarks of model performance with long context across a range of tasks, like finding a short phrase in a long document, or linking together references to the same thing, or counting occurrences of distinct words. Experiments show that models, including frontier models, can struggle with each of these tasks as the context grows past the effective context length towards the context length of the model. Take, for example, an experiment by researchers at Chroma, with a long-mem eval benchmark, which measures how well a model understands facts, across a long conversation. Each prompt contains about 500 messages between a user and a model, ending with a single question about one part of it. The model's task is to locate that part, and answer the question, to isolate the effective input length. Researchers compared two versions of each prompt. In the full version, there's all 500 messages, around 120,000 tokens in total, but in the focused version, there's less than 5 messages, around 300 tokens in total. The focused version keeps only what's needed to answer the question. Now, with the same question, and the answer is present in both, the two versions differ only in length, but models score higher on the condensed version, such as the anthropic models pictured here, meaning performance drops as the context grows, even though the task never changed. So, what's behind context rot? Partly, it's attention, and whether the model can attend to all the details in its context. Each token attends to every other token, even when they're spread out in the text. That's just how the models work. Characters and strings are transformed into numbers and arrays, and the array, representing a token, shifts, in relation to all other arrays. This is how models can understand words, in relation to their surrounding sentences and paragraphs. But, this doesn't scale well, because comparing all pairs grows with the square of the token count, meaning, at 1,000 tokens, that's about a million comparisons, at 10,000 tokens, 100 million, and at 100,000 tokens, 10 billion. Every 10 times in tokens is a 100 times in work, explaining why detail sometimes gets lost. As the context window fills up, to address long context issues, we might attempt to work in steps, each containing a subset of the context. We would break a big task, with long context, into a sequence of small tasks, with short context, well inside the effective context length. Say the context length is a million tokens. We could hold each step to 200,000 tokens. Agents could handle this, because they work across many turns of conversation, but we can't always break a task in a coherent way. For example, consider a logic puzzle. Someone makes 20 snowballs an hour, and two melt every 15 minutes. How long until they have 60? If the model has all the information, then it answers, five hours correct, but if the model gets one fact per turn, and never the whole story, then it answers too early, based on partial facts, with only the rate of making snowballs. It says three hours, because it hasn't heard about the melting yet. Or maybe it hedges, three hours if nothing melts, five hours if two do, but once the model commits to a number, it often stays stuck there, even as more information becomes available. So, without context, the model fills the gap with a guess, instead of waiting, and it happens inside the effective context length. This is a context failure mode. One of several we'll study in the next lesson, so let's discuss it there in more detail.