Sign in to continue learning
Building and Evaluating Data Agents
Intermediate
Topics
Agents
In this lesson, you'll implement your data agent as a multi agent workflow using line graph. The data agent will be able to conduct web research to answer users query and then visualize the findings or synthesize a summary of the results. All right. Let's go have some fun. In this lesson, the data agent will build will be hierarchical in nature. We'll start with the planner that will take the user's query and break it down into subgoals, ready for the agent to take action on. Then we'll pass to the executor to execute on each step, using the sub agents to accomplish each step in the plan. The sub agents in this example will have access to the web researcher, the chart generator, the chart summarizer and the synthesizer. Let's walk through an example of what a flow here might look like. add any flow. The planner will take the user query and break it down into its subgoals. First do x, then do y based on the user's query. Then it'll pass to the executor to take actions against that plan based on the goal. Then the executor will pass off to different agents to accomplish each of these steps. For example, it could pass off to the web researcher to gather and retrieve data to help answer the question. Then, after executing the web researcher, we'll go back to the executor to choose the next action according to the plan. In this example, we might then go to the chart generator to build a chart based off of the data gathered from the web researcher. We might summarize from that chart to provide a textual answer for the user, and then pass that back to the user at the end. in some cases, the data agent might also choose to do a replay. Let's take a look at that flow as well. Just like before, we'll start with the planner, where the planner will write its initial plan based on the user's query. Then again, it'll pass to the Executer to choose which agents it should use to accomplish the goals in the plan. We can use the web researcher to accomplish the first step in the plan. Pass it back to the executor. The executor might notice that it's gotten additional information from the web researcher that indicates it should replan. In this case, the planner can adjust the plan it needs to accomplish the goal. From here. Our new plan will be then provided back to the executor, which may go back to the web researcher again for a different set of information needed to accomplish the user's goal. Then we can proceed just as we did before, back to the executor, which will then pass to the task generator which will then pass to The chart summarizer and pass our textual response back to the user. Now let's take a look at how to build this agent in action. Now, we've landed in the notebook, ready to build our agent. The first thing we'll do is load our environment. This will give us access to all of the keys. We need to call a LMS, web search and other services that we need later on. Then we'll go about initializing the agent state. The agent state is what? Get the agent memory here. We'll initialize a custom state class that subclasses link graphs message statement by subclassing message state. We automatically get a messages key that will keep track of the conversation history between the different agents. In addition to these messages, we'll also include the user query, a set of enabled agents that be a planner and orchestrator will have access to. We'll also store the current step in the flow along with the agent query, which is the exact query that a particular sub agent will use. And we'll store the last reason why that sub agent was chosen by the executor. Last, we'll store some information to keep track of any Re plans to make sure we don't replan too many times. Next we'll create our planner. We can start by examining the prompts for the planner, so we can really understand how the planner will operate. to access the prompts you can go to props Dot Pi in your lesson folder. these prompts are provided to you and will help us plan and orchestrate our agents and call different agents as we use them to find the planning prompt. We'll scroll down. To the method called Plan Prompt, and we can read exactly the prompt that we're providing to the planner. We tell it that it's the planner in a multi-agent system. It's told to break the user's request into numbered steps and decompose that query into subqueries. These upgrades should be the smallest possible so that each subqueries answerable with a single data source. We'll give it an example as well. For example, if we asked what were the key action items in the last quarter and what was a recent news story for each of them? We'll give an example that it can break the BigQuery into smaller steps. You can first fetch the key action items in the last quarter, It can then fetch recent news stories for each of them. Then we'll provided the list of agents that it can call and incorporate into its plan, and we'll provide the final format that we expect the plan to be in. The final format of the plan will be a Json with enumerated steps, Where each step includes a name of an agent, along with a description of the action that that agent should take. The last important piece here is the replanning. If we are in a replan, then we'll ask the plan to provide a reason that the replan is needed, and we'll also give it some guidance to really focus on unblocking the workflow rather than finding the perfect path and to prefer a simpler, more achievable alternatives over really complex rewrites of the plan. Let's go back to the notebook to build this prompt into our planning node. As we go about building our planner, we'll first import the plan prompt that we just covered in the prompts.py. Then we'll import some things from line graph and link chain. We'll use to build our agents such as command human message and chat. Open AI to call LMS. For our planner. We'll use a reasoning LM. In particular we'll use O3 and we'll ask it using the model keyword arguments to respond. And the response format of a Json. Now we can build our planning node. So our planner node will take in the state and then respond by writing to the executor to take action on that plan. The first thing we'll do here is we'll call our reasoning model with our planning prompt. This planning prompt will be filled in by the state with the user's query. and then we'll receive an alarm reply right here that will contain our plan. We'll validate that the plan is the Json that we expect. Right here. And then you'll store as the current plan. From the reply we'll validate the Json and extract the plan using LM reply dot content right here. Then we can store as the current plan and set the replaying flag to false. In case we in a replan. In every node will return a command with two key pieces. The first is update. Update will update the plan messages containing the message history between the sub agents will continue to retain the user query. Will retain the current step in the plan, along with the last reason, we'll continue with the same list of enabled agents, and we'll also keep a replan flag. When we write update into the command this is updating the state. So you'll notice a lot of the same keys in the state that we had when we defined it. To start with you'll see the plan, the messages, the users query and more that we all started with in our initial state. And we're just updating it here. The second piece in the command is to go to the go to tells us what sub agent should be called next. And because we're in the planner, we'll move from the planner to the executor indicated here in the go to. Now we'll create the executor. We can best understand the executor, just like the planner, by first looking at its prompt. let's take a look at the prompt. Now. We're turning back to prompts dot pi. We can find the executor prompt right below the planner prompt. We'll just scroll down. Into the executor prompt. The first thing we tell the agent is that it's an executor and a multi-agent system. With the following agents. And we'll extract the list of agents from the enabled agents list in our state. We'll then provided a series of tasks. First it needs to decide if the current plan needs a revision. It needs to decide which agent to run next and why. And then it needs to write the exact question that the chosen agent should answer. This is the sub question that we've been talking about. We'll also give it some guidelines on how to handle plans and we'll tell it to respond. Invalid Json, including the keys such as replan. Go to the reason and the query. Then we'll give it some guidance to prioritize forward progress. We'll tell it how to choose the go to. in this step we'll ask if the agent has made reasonable progress, and if so it can move to the next steps agent and then an. Otherwise it should execute the current steps assigned agent. Then we'll give it instructions on how to build the query. We'll tell it to write a clear standalone instruction for the chosen agent. That standalone question should be written in plain English and answerable by the agent. Last, we'll tell that the context that has to make these decisions. We'll give it the user query, the current step index and current plan step. We'll tell it if it's just preplanned and we'll give it the message history. Then it's ready to make its decision. Let's go back to the executer in the notebook so we can see how we use this prompt in action. So the first thing we'll do as we create the executor is will import the executor prompt that we just defined in prompts dot pi. And then we'll also import the and type from Lin graph. Now, I'll set the number of max three plans for my agent. I chose the maximum where he plans of three so that the agent doesn't get carried away in its replanning. Then we can start to define our executor node. Our executor node is going to take in the state which contains all of the information that we just talked about. And it's going to return a command that routes to the next step agent. This could be the web researcher chart generator, synthesizer, or even the planner. Then we'll extract the plan and the step from this state. The first thing we'll do here is we'll then build our executor prompt, just like we did in the planning node. And we'll call the reasoning Elm with this executor prompt. Once we've received the response we'll extract the execution in the content. And then we'll pass it to get out the plan. The go to the reason and the query. Once we've done that, we can update the state with this information. If we've chosen to replan, we need to account for that here. Here we'll retrieve the replan attempts from the state. If we've done some replanning, we need to account for that here. We'll get the replan attempts from the state and we'll set the step, replan. And then we'll decide if we have replan. We'll make sure that we haven't exceeded the number of max allowable plans. We'll iterate the Re plans by adding one two. Step three plans. And we'll update the state again right here. If we are doing a Re plan and the step three plans is below the max number of Re plans, we'll return command and go to the planner with these updates. And then otherwise we'll go to the synthesizer or the next planned agent. Last. If we're not re planning will activate the next plan agent by getting the agent from the current step. We'll set the updates. Current step. Add one. And then we'll set the re plan flag equal to false. And we'll return the command with any of the updates we've made to the state. And the chosen agent in the go to. We've missed one piece is that if we have just re planned, we need to run the planned agent once before you consider. Let's add that now. So how will determine. This is we'll look at the state and get the re plan flag at the re plan flag. It's true. Then we'll discover the plan agent and get that agent. And we'll run that plan to agent by routing in the go to. We'll also iterate the current step and set the re plan flag to false. In this case. now that we've created the planner and the executor, we can go about creating our sub agents. These sub agents will be used by the executor. If you take actions against the plan and reach the user's goal, the first sub agent will define is the web researcher. This is going to do the bulk of our data tasks, getting information from the web ready to generate charts and synthesize and respond to the user with an answer. Let's take a look at how we create that web researcher. To start building our web search agent. We're going to use a react agent from link graph. A link graph react agent will use a tool along with an Elm to accomplish its task. This will be the basis of our SAP agents here. The tool we'll use for the web search agent is Tadley will import search from link chain to heavily. And then we'll define our tool and we'll pass in a max results to five. This is the number of results we'll get from the open web. To see what this does in the tool call alone we can invoke the tool call directly just like so. Here we'll ask what JPMorgan's current stock price and return the results. And the results we can see a list of URLs, titles and the content, along with some other information for each result. this is what the react agent will use to answer the question. This would be a lot of information to pass back to our master multi-agent system. So by using a react agent, we get to actually provide only the information that the agent system needs to answer the sub question provided to that sub agent. Now let's build our react agent using this tool. to build our react agent. Here we'll import the agent system prompt from the helper module and set the LM to GPT for our. Then we'll create the web search agent by calling Create React Agent, passing in our chosen lamp and tool, the tablet tool, and then providing a prompt in this prompt, we tell it that it's the researcher and that it can only perform research using the provided search tool. Tablet tool When it's found all of the needed information, it should end its output and not take further actions. Executing the cell, we'll initialize our web search agent and then we can try it out. Now we can execute our web search sub agent with the same query we did before, and ask JPMorgan's current market cap. By using a web search agent armed with the tool, rather than the tool alone, we could provide only the needed information back to our agenda system. Now we get a nice, clean answer containing exactly what JPMorgan's current market cap is without any extra information that we don't need. The last thing we'll do when we define our web search agent is to wrap it in a line graph node. This allows us to build it into a graph. The web search node will take in the state just like every node, and it will also route the executor. That way, the executor can choose the next action in the plan. After the web search node. Then we'll retrieve the agent query from the state. This is the exact sub question that's being passed to the web search agent. Then we'll invoke our web search agent, just like we did before, using this message as key and passing in the agent query. Last, we'll wrap this response in a human message. This is the type that will allow us to wrap this response in a humored message type from Link Chain, And we'll go back to the Executer using the command along with updating the state. here. We'll return command containing an update to the state and the next action agent to take in the update to the state. We'll update the messages which tracks our conversation history between our different sub agents, with the result from our web search agent. The resulting messages from the web search agent get appended onto the conversation history in the messages key. In our state. Now we've created our web search agent. Next we'll go to the chart node. All right. We just finished creating our web search agent. This is going to do our data tasks for this agent system. Now we can create our charting agent, the next sub agent in our architecture. our target agent, just like the web search agent will be a react style agent. That means we're binding the LM to a tool. But in this case, rather than a web search tool, we'll use a Python rappel tool. This is gonna allow the LM to execute Python particularly. It's going to allow the LM to execute Python in order to generate a chart. We'll give it the prompt that it's tasked with generating charts and working with a researcher colleague. We tell it to print the chart first and to save it to our current working directory, and then to tell the chart summarizer exactly where the path is, and some notes summarizing the main insights. We'll also wrap this agent in a line graph node. This agent again is going to take in this state, tracking all of the same information in the agent's memory. And it's going to route into the chart. Summarizer. The first thing we'll do is we'll invoke the chart agent using the state. And then we'll take out the content and wrap it in a human message. These messages are going to be passed into the state and appended to our conversation history tracked between the sub agents. As we mentioned in the tip end. We'll also go to the chart summarizer using the command. Following from the chart generation agent. We'll create our chart summary agent. The chart summary agent will take the image, the chart itself, and provide a nice textual response back to the user. similar to the last sub agents. We'll also create a react agent here, but this time we won't use any tools. We'll also provide a prompt that tells us to generate image captions. And it's tasked with generating a standalone, concise summary of the provided chart. Save data local path. Then we can wrap our react agent in a chart summary node. The chart summary node is going to invoke our react agent and then pass to the end. The last subject we need to create is that the synthesizer agent. This helps with the case that the agent is not tasked with generating a chart, and instead just needs to respond with a textual response. the next agent. We'll create his art synthesizer agent. This synthesizer agent is going to directly invoke our alarm with a provided prompt, so that we can take in the entire information that the agent has already gathered and provide a response back to the user. Well, first, start by extracting the relevant messages from the conversation history, by sorting through the messages and pulling out any from the web searcher, the chart generator, or the chart summarizer. For a synthesizer, we'll again use GPT four zero as our alarm. And then we can create our synthesizer node. Our synthesizer node is going to gather the informative messages that it needs for the final synthesis. We'll do that by extracting the message history from the state, and then finding any messages from the important agents the web researcher, the charge generator, the target summarizer. Notice that this doesn't include the planner or the executor. Then we'll fetch the user question from the state as well. That way the synthesizer is armed with the exact user query it needs to answer. Once we have this information, then we can create our synthesis instructions. This is the exact prompt we'll pass to the Elm to tell it how to answer the question. This is very stylistic. We tell it to perform any lightweight comparisons or infinitives as required, but not to invent any facts not supported by the context. We'll tell it to provide a concise response that fully answers the question, and to start with a direct answer. And including any citations if it needs. And to keep the output crisp. Now that we have our relevant messages, the user question and the synthesis instructions, we can build our prompt containing each of these key pieces of information. Our summary prompt is going to be a human message with the content of the user question, followed by the synthesis instructions, and then last followed up with the content from all of the relevant messages in that conversation history that we've been tracking in the state. Once we've built, our prompt will invoke our alarm with this prompt. Exactly. When we received the LLM reply. We'll extract out the content and then we'll print the synthesize our answer just so we can see it. And we'll finish off. By routing to the end and updating the state with the final answer. And our last set of messages. Now that we've created each sub agent in our graph, we can start to build the agent graph. Let's remind ourselves of the architecture we've been building. So now we've created all of the nodes we need to finally build out this workflow. We started with the planner and then the Executer, and then we built out each of the sub agents needed to accomplish goals according to the plan. We created the web researcher, the chart generator, the chart summarizer, and the synthesizer. Now we can build the graph. To do that we're going to import state graph from line graph. And we're going to instantiate it with our custom state. Then we'll add each of the nodes to our workflow that we've been creating. In this lesson. Last we'll add a single edge from the start to the planner so that we always start with a plan. Now we can compile our workflow into the variable named graph. To verify that we have created the graph we attend. We can draw a mermaid chart from this graph using a provided link chain utility. This gives us exactly the architecture we intended. It looks a lot like the slide we saw before, doesn't it? Now that we've created our graph, we're ready to go and ready to start using our agent. So the first question we'll ask the agent is to chart the current market capitalization of the top five banks in the US. We expect this to rely on web search to capture the current market cap of these banks, and then go to the chart node to generate a chart, and then last the chart summarizer provide a textual companion response, we'll invoke that graph with the state, including the user query. We'll start off our messages history with that user query right at the beginning. And then we'll also provide a list of enabled agents. These enabled agents are slotted into the planning prompt so that the planner only writes a plan with the available agents and the Executer prompt, so that the executer only chooses agents available and ready to use. So our agent has responded with an answer. It's provided us a nice chart with the top five market caps of U.S. banks with JPMorgan in the lead. the charts. Firm riser, however, did not do as good of a job. It just provided us with the chart path, but didn't actually give us the textual response we're looking for. In other cases, the alum will actually give the full textual response, and you might see that in your notebook. But either way, we'll observe exactly how well this agent is doing when we get into evaluations later. Let's try a second query. In this query, we asked to identify regulatory changes for the financial services industry in the US. Compared to the last example where we were looking to generate a chart. And the chart Summarizer here we're taking the synthesizer route. We're looking to only respond with the text response. Just like before, we're going to invoke the graph with our user query and initialize messages with that query. And then the list of enabled agents that we have to work with. Let's wait for the agent to return a results for us. So our agent has responded with a nice, detailed answer to answer our question. It gives us key details about regulatory changes in Congress, the SEC, the Federal Reserve, and more. And it even provides some citations about where it got this information from. in the next lesson. We'll go on to expand our agents capabilities and allow it to reason over internal data as well using text, SQL and document search. These will be additional sub agents provided into our system so that we can reason and answer even harder questions and provide more value to our end user. Let's get to the next one.
Building and Evaluating Data Agents