Sign in to continue learning
Building toward Computer Use with Anthropic
Beginner
1h37m
Topics
AI Coding
AI Safety
Agents
Chatbots
GenAI Applications
Generative Models
MultiModal
Prompt Engineering
Task Automation
By the end of this lesson, you'll be able to understand tool use workflows with Claude. You'll define your own tools and implement your own tool-based workflows from start to finish. Let's get to it. So let's start with an explanation of what tool use actually is. Tool use, also known as function calling is basically the ability to extend Claude's inherent capabilities. We can give Claude some set of external tools or functions, give it access to those functions, and then allow the model to call those at any point, although to be a little bit more specific, the model itself is simply requesting to call a tool or a function, and then we actually do the invocation. We actually do the function execution and then send the result back to the model. So we'll see some diagrams and we'll build an example of tool use, a little chatbot for customer support step by step. But before we do that let's talk about why this matters. The short story is that tool use vastly extends the use cases for Claude models. We can take our normal prompts, whether they're image-based prompts or text prompts, and combine them with calls to external functions. And again, Claude itself, or the model itself is not directly calling the tools, but it's deciding that it wants to call a tool. It's passing arguments through. And then we call actual tool and then pass the result back to Claude. And this enables us to do all sorts of things that the models can't do on their own, like retrieve dynamic data, query our own databases internally, interact with APIs, execute code, search the web, and of course, control a computer. As you'll see at the end of this video, tool use is what powers and enables computer use. The model can output tool calls to click or move a mouse or type, and various other computer actions whenever it wants. And then we can execute those actions and tell the model, all right, we did the click you wanted or we did the mouse move that you wanted. Here's what the screen now looks like. And give it a new screenshot. We'll build up to that. To start we need to understand how tool use works. Back in a notebook we have the same basic setup from previous lessons. This diagram shows the basic flow of tool use. On the left, right here, we have our model. We give it access to a set of tools. In this case inventory lookup, generate invoice, send email. Maybe these are things you want the model to be able to do. And in this particular diagram the model decides it wants to call the tool called inventory lookup. So step one, the model says I want to call this tool. And we'll see what this looks like in a bit. Then it's up to us or up to you to actually execute that tool, that function. So inventory lookup with some arguments presumably is going to connect to a database or some sort of API to do some lookup in our inventory somewhere. Then once you get some results back, you send them off to the model, and the model can complete whatever process it was trying to do. So again, the model issues a request to call a tool. Then you have to execute the tool. Then you tell the model about those results. And then the model can carry on and do whatever it wanted to do next. This more extensive diagram details the process. You give the model a prompt and a set of tools. You'll soon see how you do that, how you actually write the set of tools and provide them to the model. Then the model may decide it wants to use a tool, so it outputs that. It says, I'd like to use its tool with these arguments. Then you actually execute the tool. You get some result a return value or data from an API, or data from a search tool, or from a database. Then you send a tool result back to the model saying, here is the result of the tool you wanted to call. And then finally the model can do something with that result. So there's a couple of steps involved here. First, you need to have some set of tools you want to give the model. Second, you have to define them in the proper format and tell the model about those tools. And then third, you have to understand the back and forth flow and the proper format for returning a tool result to the model. To start, I'm going to show you a fake database class that we will be using in place of a real database, just to keep things simple. Imagine we run a company called creatively Acme Corporation or Acme Co, and this Acme Corporation has a few different products. We have customers. We have orders. So as you can see here, customers, it's all just placeholder data just to keep things simple. But someone like John Doe has an email and they also are going to have a username, John Doe. There's an orders list in this class representing the database. Each order has an ID, it has a customer ID, and it has a product quantity a price, a status, whether it's been shipped or if it's processing. Very, very simple. Just taking the place of a real database to keep things simple. This class has various functions or methods associated with it like get user, get ordered by ID, get customer orders, and cancel order. You can instantiate fake database here and imagine that you are a customer support representative. And somebody chats with you and says, my name is John Doe. My email is [email protected]. I want to cancel in order. If you look at the orders, they don't have John Doe associated or any username associated with them. They have a customer ID. Each user has a customer ID, and if you look at the methods we have available, we have get user, which will just find a user based off of an email, a phone or a username. We have get order by ID which requires an order ID. have get customer orders, which requires a customer ID, and then cancel order, which requires an order ID. So if John Doe wants to cancel an order, it's not as simple as just cancel order unless John Doe happens to know his exact order ID. So there's multiple steps involved. The first step as a human is to get user based off of this email. [email protected]. Then you can see that John has this user ID. The next step is to take that ID for John Doe find his orders using get customer orders. And then if you are acting as a customer support representative, you would likely follow up and say, I see you have three orders. Two of them have been shipped. We can only cancel a order that has not been shipped yet. Do you want to cancel the smartphone case order? In which case you could finally take this order ID right here and cancel this order using DB dot cancel dot order. And that will cancel that order. Okay, so multiple steps involved is the point here. It's all fake, right? Not a real database. But imagine you do run a company with a real database. And you want to use Claude to help automate portions of customer support. Maybe the low-hanging fruit. And you need to give Claude access to these functions. It needs to be able to look up a user by an email or a phone or a username. And instead of ten users or whatever we have here. Imagine there's tens of thousands and needs to be able to find orders and needs to be able to find orders based on a customer. It needs to be able to cancel orders. Just a small set of tools here. So how do you tell Claude about these tools? First step is to define the schemas for each of your functions. And the way that you do that is through Json schema. If you're not familiar with Json schema, it's a common open format that you can use to define well schemas. And in this case, Claude expects or the API expects tools to be passed in using this Json schema format. This is a hypothetical tool definition for the get user method you saw previously. So it has a name, get user, a description. And these tool descriptions are important to tell. Claude "Here's what this tool does" And then an input schema that includes the arguments necessary to call this tool. As a reminder, this is what get user looks like. So the underlying function has two parameters: key and value, both of which are required. Going back to the tool definition for get user, we write the schema to indicate that there are two arguments. One is called key. One is called value. Both are strings. Key is one of the following choices using this enum, which is a nice syntax to say that it has to be either email, phone, or username. The idea here is that you can look up a user by an email, or by a phone number, or by a username, and there's also a description attached explaining how it works. And then also a value which is a string which needs to be the actual email like John @gmail.com, or the actual phone number, or the actual username. And then finally down at the bottom here you can indicate which of these arguments are required to call this tool. In this case, both of them are required. So the next step is to define all of these tools. We have four different functions. We'll have four tool definitions or tool schemas using Json schema and put them in a list called something like tools. The name doesn't matter, but we'll group them together. This list, called tools, contains four Json schema tool definitions. Get user, get ordered by ID, get customer orders, and cancel order, all of which correspond to the previously defined methods we saw in our fake DB class. The next step is to give Claude the tools and send a prompt. So this is just a simple one off example. Not in a function, not in any sort of chatbot loop. You have a messages list with a single message from a user saying show me my orders. My username is priya123. Then most importantly, this highlighted line. In addition to the fields we've seen before or the parameters you've seen like model max tokens and messages. Now you're passing through tools. Tools is the list of tools previously defined. The model will know it has access to these tools and can decide if it needs to call any. Presumably, if this prompt is asking to show, you know, show me my orders. My username is priya123. The model should call a tool to look up orders based off of this username. Execute the cell and then print response dot content. Now we have something we've yet to see. There's a text response. I'll help you look up your orders. Let me first get your user details using your username. And then I can retrieve your orders. Followed by something called a tool use block. So we have a text block and the tool use block. And this tool use block, if I just let's take the last element in this list so we can hone in on that. The model wants to call this tool which is called Get User. Specifically it wants to call it with the argument key set to username and value set to priya123. There's also this tool use id which is important because the second step once the model says it wants to call the tool, is for you to actually execute the tool. Then tell the model what the result is. And when you do that, you give the model the tool result itself, the data from the function along with this ID, so that the model knows how this tool result corresponds to the initial tool use block. The next step is to translate the model's ToolUseBlock output, basically stating I'd like to call get user with these values and actually execute get user with those values, or execute any of the other tools, get order by ID or cancel order, or get customer orders. This is a very simplistic implementation, but that's what this function does. It takes a tool name and a tool input. And this is tool input right here. Then it's as simple as taking the tool name and the tool input that the model outputted for you. And then pass them to process tool call to actually execute that function, which is using the database, the DB that you saw previously. And that's the first half. The second half is to then send the result back to the model. To do that you need to send a follow-up message with a role of user. Where content has a type of tool result. Tool result must contain the actual content. In this case, this is the content back from calling a tool and the tool use id. This ID needs to match the corresponding tool result that you send back to the model. Simply defining this, of course, won't do anything. You need to send it to the model and have some sort of loop or some function so that you can continue chatting or conversing with the model. That's the next step is to put it all together so you can see the complete flow. This is a simple, complete implementation of a chatbot that uses the four tools you saw previously. There's a lot to cover. You don't need to go over it in extreme detail. It's there for you to look at on your own. But this is the core functionality, this function called simple chat. It uses a system prompt, which is a way to provide, repeated instructions to set the role, sort of set the expectations of the model. So in this case, the prompt says your customer support chatbot for Acme Co, your job is to help users look up their accountant orders and so on. And then in each conversational turn, begin by thinking about your response. Once you're done, write a user-facing response. This allows us to separate out the model's internal thoughts versus the actual reply that we want to show to a user as a chatbot. It's important to place all user-facing conversational responses inside of reply XML tags to make them easy to parse. A note about that. There's a function called extract reply that does exactly what it sounds like. It extracts the contents between reply XML tags. Further down, we start by asking for user input. There's a messages list that starts with a single user message. Maybe it's just "hello." There's a way to break out of the loop. If the user types the word quit and then it's all about tool use, the messages are sent off to the model in this loop. All four tools are provided as that list of tools previously defined. The prompt. The system prompt is provided. Model name Max tokens. All of that that you've seen before. Then the model responds with something. You append the message that the assistant responded with. Remember, there's this list of messages constantly growing when you're trying to build a chatbot. And most importantly for this lesson, if the model stopped because of tool use, if you recall, you saw stop reason a few lesson to go. One reason the model stops is because it just reached a natural stopping point. It's just it's done. Another reason is that it hits a stop sequence. Another reason is that it hits max tokens. And another reason is that it wants to call a tool so it stops generating. It's waiting to get a tool result. If that's what the model wants to do, then extract the tool use information including the name and the input. There's a print here that just lets you know that Claude wants to use a particular tool, then process the underlying tool using the function you saw previously, Process tool call. Finally, append a tool result to the conversation history. That includes the tool use id and the actual value returned from the underlying tool function and then the loop repeats. If Claude didn't want to use a tool, then simply print out the model's response and then you can try chatting to the model. Simply run the simple chat function and then enter your first conversational message. Maybe something like "hello", to start. There's a response. "Hello, welcome to Acme Co customer support. How can I help you? I can look up your account info, check order status or cancel orders." Let's say I'll post as Priya, which is one of the users in the fake database with the username of priya123. So I'll write that I want to check on my order status and then the model now says. "I'd be happy to help you check on that. Provide either the order ID", I don't have that. So instead I'll say I have my username which is priya123. Hit enter and the model will likely want to call a tool now. It wants to call Get user and it actually called two tools. The first time it called to get user to look up information about Priya. That then gave previous user id. The model then called get customer orders using the user id from Priya. Finally, it found the orders and gave a response saying "you have two recent orders a Bluetooth speaker and wireless headphones. Is there anything specific you want to know about those?" So that's a simple demonstration of the workflow. To wrap up, we'll return to our computer Use Quick Start implementation. We'll take a much closer look at this in the next video. But I just want to show you that it does in fact use tools. There's an import at the top to import various tools. A computer tool is the one that's most interesting to most people. It includes clicking functionality and mouse movement. All the tools are combined into this tool collection variable. When a request is sent to the model, the tools are passed through. The same thing you just saw tools equals some list of tools. And then once the model's responded back, if it wants to issue a tool or use a tool if the content block type is tool use. There's logic in here to execute a tool. And then make a tool result and send that back to the model. Slightly nicer code. A few helper functions. Things are split across files, but it's the same core tool use functionality that you saw in this lesson.
