Sign in to continue learning
Context Engineering
7h55m
In the last lesson, we discussed the need to build up context for our agents. We can wire together a harness around the model to retrieve, manage, and store information for tasks. The harness turns the model into an agent that can run code to handle different components of context, like prompts, memory, traces, and tools. While we can add various parts to a harness, the harness must contain tools, because without tools, a model isn't an agent. Without tools, a model might respond to prompts about asking, but not prompts about doing, and even prompts about asking. Say, asking for current information or specialized information won't be possible for a model without access to tools. So, let's better understand the details of tool calling in the harness, while context engineering involves retrieving and organizing and storing information. Retrieving information is especially important. Otherwise, there's nothing to manage or store, and in my experience working with practitioners, getting retrieval right will later help getting organization right and storage right. Tools provide additional context to the model, because tools are remote procedure calls for the model. Think of them like functions that the model needs for, searching the web or querying a database or reading a file. For example, the agent could read content from a file, say ATX file or MD file, through a tool called red file, which is one of several tools in the harness. Alongside maybe query database and search web, the implementation of the read file tool could be a Python function that takes a file path as input and provides the contents of the file as output. Since the string is available in the harness, the agent can use that data. Granted, it's relevant for the model, but it's not your responsibility to determine the relevance of that data, whether it's selecting a tool, setting the inputs of the tool, or getting the outputs of the tool. All that must be determined by the model. So, let's understand selecting, setting, and getting in detail. First, the model must select a tool from several tools available in the harness. If the model decides that it cannot provide an adequate response to the user, then it must compare the user prompt to each of the tool descriptions. The tool descriptions are part of the string representation of the tools available in the context window of the model. Commonly, the string representation has a JSON schema with the fields name, such as read file, description, such as reads the content of a file, and parameters, where parameters contains names and description for each tool input, such as file name, which is the path to the file as a string. Remember, the model has no memory, so the context window must contain a list of tool descriptions. At each step of the agent's trajectory, the string representation of the tool doesn't contain the code implementing the underlying function. Note that the code from the harness isn't here in the context window of the model. Instead, you have explanations of the tool's usage, and alongside it, limitations of the tool, like errors that might be thrown at execution, or maybe the relationship to other tools. Maybe a few tools have to run together in a particular order, triggered with details about parameters. For example, if the user asks about inflation from last month, then the model cannot provide a response with up-to-date information. The agent has to select a tool, according to its description, from the harness. Here, it chooses search web over query database and read file. Additionally, the model must provide values for the inputs of the tool. Think, populating the arguments of a function with numbers or characters related to the user prompt, often extracted from it. Based on the string representation of the search web tool, the model knows about each parameter of the underlying function, meaning it sets a value to something like the string, US inflation last month. Remember, the model doesn't run the code in the implementation of the tool, such as the search web tool. The harness has to run the code of the underlying function for web search. The response is the tool output. Note that tools and tool outputs are different. Tools are remote procedure calls for the model. They appear in the context window through their string representation, containing name, description, and parameters. Tool outputs are the data provided by running the tools. They appear in the context window as characters or numbers, depending on the shape of that output. Suppose the search web tool provides a list of URLs. Since the input was US inflation last month, we might expect a few URLs in the response, including the website from the US Bureau of Labor Statistics, which contains historical inflation rates, as measured by Consumer Price Index, namely CPI. Having retrieved a useful link to a website, the agent needs to fetch data from the website. Thankfully, the agent can call tools in a loop. Let's suppose we have another tool in the harness called FetchWeb that takes a URL as its input, sends a GET request to the corresponding website, and provides the HTML response as its output, since the model has the URL to the Bureau of Labor Statistics website in its tool output from the search web tool. It can decide to select the FetchWeb tool with the indicated URL as input. The agent can have multiple iterations in its loop of tool calling. At each iteration, the model must decide either to provide a response to the user or to select another tool in an additional iteration. The harness implements the loop in code. Think a while loop that keeps running, as long as the model selects another tool. Here, the tool output from FetchWeb will be lots of data from the web page. The raw response might be cluttered with lots of irrelevant information, like HTML tags. All those angular brackets aren't that informative. Actually, the model might mistake them for special characters in the prompt, added for emphasis on certain phrases or organization between different paragraphs, Rather than pass the tool output directly into the context window, the harness could reformat the tool output, removing that irrelevant information by stripping out whitespace and tags from the HTML. For example, the harness might have a function called stripHTML, which processes down the context from the web page with some regular expressions. The stripHTML function could be a tool, like search web or FetchWeb, but the agent would need another iteration in the loop of tool calling to employ the function. It would be faster and cheaper to use the stripHTML function as a hook, instead of a tool. Hooks are snippets of code that run in the harness on different triggers. Commonly, these triggers are events like before calling a tool or after calling a tool. Here, the trigger for stripHTML would be the FetchWeb tool. In practice, there are lots of events besides tool calling, especially around user actions, like sending a message or receiving a message. Hooks let you add monitoring and content moderation and many other components to the harness, without asking the model to do it itself. The same considerations apply to the stopping condition of the loop. Either the model stops the loop, or the harness stops the loop. As mentioned, the harness implements the loop of tool calling, and like any loop, there needs to be some stopping condition. The stopping condition could depend on the model, where the loop runs until a break command, whereupon the model decides not to call a tool, or the stopping condition could depend on the specifications in the harness, where maybe there's a maximum number of allowable iterations. We can understand the nickname harness better now. The loop repeats, often many times, until the model has enough information to yield a final response to the user. But if the loop goes on and on, the harness needs to rein in the model through a hard stop. Here, we obtained the right response of 2.2%, because the agent selected the right tools, searchWeb, then FetchWeb, set the right inputs, the query, then the URL, and got the output from FetchWeb tool, reformatted by the strip HTML hook. That all came together, but in practice, there can be challenges with tool calling, like calling the right tool with the wrong inputs, maybe interest rates, not inflation rates, or calling the wrong tool, maybe calling FetchWeb on the query, or searchWeb on the URL. Commonly, these mix-ups happen when there are similar names, or descriptions among tools, or not calling a tool at all. The model might rely on its own knowledge from training, as opposed to its tools, which allow it to retrieve additional information. To avoid these challenges, we need to implement tools carefully in the harness. In the next lesson, you will build your own agent with a tool to retrieve information from the file system. We'll see all of this in action, so join me there, and let's get coding.