Sign in to continue learning
Building and Evaluating Data Agents
Intermediate
1h59m
Topics
Agents
In this lesson, you'll implement your Data Agent as a Multi-Agent Workflow using LangGraph. The Data Agent will be able to conduct web research to answer user's query, and then visualize the findings or synthesize a summary of the results. All right, let's go have some fun. In this lesson, the Data Agent we'll build will be hierarchical in nature. We'll start with the Planner that will take the user's query and break it down into sub goals, ready for the agent to take action on. Then, we'll pass to the Executor to execute on each step using the sub-agents to accomplish each step in the plan. The sub-agents in this example will have access to the Web Researcher, the Chart Generator, the Chart Summarizer, and the Synthesizer. This helps with the case that the agent is not tasked with generating a chart, and instead just needs to response with a textual response. Let's walk through an example of what a flow here might look like. In any flow, the Planner will take the user query and break it down into its sub goals. First do X, then do Y, based on the user's query. Then, it'll pass to the Executor to take actions against that plan based on the goal. Then, the Executor will pass off to different agents to accomplish each of these steps. For example, it could pass off to the Web Researcher. to gather and retrieve data to help answer the question. Then, after executing the web researcher, we'll go back to the executor that choose the next action according to the plan. In this example, we might then go to the chart generator to build a chart based off of the data gathered from the web researcher. We might summarize from that chart to provide a textual answer for the user, and then pass that back to the user at the end. In some cases, the data agent might also choose to do a replan. Let's take a look at that flow as well. Just like before, we'll start with the Planner, where the Planner will write its initial plan based on the user's query. Then, again, it'll pass to the Executor to choose which agents it should use to accomplish the goals in the plan. We can use the Web Researcher to accomplish the first step in the plan. Pass back to the Executor. might notice that it's gotten additional information from the Web Researcher. that indicates it should replan. In this case, the Planner can adjust the plan it needs to accomplish the goal from here. Our new plan will be then provided back to the Executor, which may go back to the Web Researcher again for a different set of information needed to accomplish the users' goal. Then we can proceed just as we did before, back to the Executor, which will then pass to the Chart Generator, which will then pass to the Chart Summarizer and pass our textual response back to the user. Now let's take a look at how to build this agent in action. Now we've landed in the notebook, ready to build our agent. The first thing we'll do is load our environment. This will give us access to all of the keys we need to call LLMs, web search, and other services that we need later on. Then, we'll go about initializing the agent's state. The agent state is what gives the agent memory. Here, we'll initialize a custom state class that subclasses LangGraph's MessagesState. By subclassing MessagesState, we automatically get a messages key that'll keep track of the conversation history between the different agents. In addition to these messages, we'll also include the user_query, a set of enabled_agents that the planner and orchestrator will have access to. We'll also store the current_step in the flow, along with the agent_query, which is the exact query that a particular sub agent will use, and we'll store the last reason why that sub agent was chosen by the executor. Last, we'll store some information to keep track of any replans, to make sure we don't replan too many times. Next, we'll create our planner. We can start by examining the prompts for the planner so we can really understand how the planner will operate. To access the prompts, you can go to prompts.py in your lesson folder. These prompts are provided to you and will help us plan and orchestrate our agents and call different agents as we use them. To find the planning prompt, we'll scroll down, to the method called plan_prompt. And we can read exactly the prompt that we're providing to the Planner. We tell it that it's the Planner in a multi-agent system. It's told to break the user's request into numbered steps and decompose that query into subqueries. These subqueries should be the smallest possible subquery so that each subquery is answerable with a single data source. We'll give it an example as well. For example, if we asked what were the key action items in the last quarter and what was a recent news story for each of them? We'll give it an example. can break the big query into smaller steps. It can first fetch the key action items in the last quarter. It could then fetch recent news stories for each of them. Then, we'll provide it the list of agents that it can call and incorporate into its plan. And we'll provide the final format that we expect the plan to be in. The final format of the plan will be a JSON with enumerated steps where each step includes a name of an agent along with a description of the action that that agent should take. The last important piece here is replanning. If we are in a replan, then we'll ask the plan to provide a reason that the replan is needed, and we'll also give it some guidance to really focus on unblocking the workflow rather than finding the perfect path, and to prefer simpler, more achievable alternatives over really complex rewrites of the plan. Let's go back to the notebook to build this prompt into our planning node. As we go about building our planner, we'll first import the plan_prompt that we just covered in the prompts.py Then we'll import some things from langgraph and langchain we'll use to build our agents, such as Command. HumanMessage and ChatOpenAI to call LLMs. For our planner, we'll use our reasoning LLM. In particular, we'll use o3, and we'll ask it using the model_kwargs to respond in the response_format of a JSON. Now, we can build our planning node. So our planner_node will take in the state and then respond by routing to the executor to take action on that plan. The first thing we'll do here is we'll call our reasoning model with our plan_prompt. This plan_prompt will be filled in by the state with the user's query, and then we'll receive an llm_reply right here that will contain our plan. We'll validate that the plan is the JSON that we expect. right here. And then we'll store as the current plan. From the llm_reply, we'll validate the JSON and extract the plan using llm_reply.content right here. Then, we can store as the current plan and set the replan_flag to false in case we're in a replan. In every node, we'll return a command with two key pieces. The first is update. Update will update the plan, messages containing the message history between the sub-agents. We'll continue to retain the user query, we'll retain the current step in the plan, along with the last reason. We'll continue with the same list of enabled agents, and we'll also keep a replan flag. When we write update into the command, this is updating the state. So you'll notice a lot of the same keys in the state that we had when we defined it to start with. You'll see the plan, the messages, the user_query, and more. that we all started with in our initial state and we're just updating it here. The second piece in the command is the goto. tells us what sub-agent should be called next, and because we're in the planner, we'll move from the planner to the executor, indicated here in the goto. Now, we'll create the executor. We can best understand the executor, just like the planner, by first looking at its prompt. Let's take a look at the prompt now. We're turning back to prompts.py We can find the executor prompt right below the planner prompt. We'll just scroll down. into the executor prompt. The first thing we tell the agent is that it's an executor in a multi-agent system with the following agents. and we'll extract the list of agents from the enabled agents list in our state. We'll then provide it a series of tasks. First, it needs to decide if the current plan needs a revision. It needs to decide which agent to run next and why. And then it needs to write the exact question that the chosen agent should answer. This is the sub question that we've been talking about. We'll also give it some guidelines on how to handle replans, and we'll tell it to respond in valid JSON, including the keys such as replan, goto, the reason, and the query. Then, we'll give it some guidance to prioritize forward progress. We'll tell it how to choose the goto. In this step, we'll ask if the agent has made reasonable progress, and if so, it can move to the next step's agent. And then otherwise, it should execute the current step's assigned agent. Then, we'll give it instructions on how to build the query. We'll tell it to write a clear, standalone instruction for the chosen agent. That standalone question should be written in plain English and answerable by the agent. Last, we'll tell it the context it has to make these decisions. We'll give it the user query, the current step index, and current plan step. We'll tell it if it's just replanned and we'll give it the message history. Then, it's ready to make its decision. Let's go back to the executor in the notebook so we can see how we use this prompt in action. So the first thing we'll do as we create the executor is we'll import the executor_prompt that we just defined in prompts.py And then we'll also import the END type from LangGraph. Now, I'll set the number of MAX_REPLANS for my agent. I chose a maximum replans of three so that the agent doesn't get carried away in its replanning. Then we can start to define our executor_node. Our executor_node is going to take in the state, which contains all of the information that we just talked about and it's going to return a command that routes to the next sub-agent. This could be the web researcher, chart generator, synthesizer or even the planner. Then, we'll extract the plan and the step from this state. The first thing we'll do here is we'll then build our executor prompt just like we did in the planning node, and we'll call the reasoning LLM with this executor prompt. Once we've received the response, we'll extract the execution and the content, and then we'll parse it to get out the replan, the goto, the reason, and the query. Once we've done that, we can update the state with this information. If we've chosen to replan, we need to account for that here. Here, we'll retrieve the replan_attempts from the state. If we've done some replanning, we need to account for that here. We'll get the replan_attempts from the state and we'll set the step_replans. And then, we'll decide if we have replanned, we'll make sure that we haven't exceeded the number of max allowable replans, we'll iterate the replans by adding one to step_replans, and we'll update the state again right here. If we are doing a replan and the step_replans is below the max number of replans, we'll return Command and go to the planner with these updates, and then otherwise, we'll go to the synthesizer or the next planned agent. Last, if we're not replanning, we'll activate the next planned agent by getting the agent from the current step. We'll set the updates current step add one. and then we'll set the replan flag equal to false. and we'll return the Command with any of the updates we've made to the state and the chosen agent in the goto. We've missed one piece is that if we have just replanned, we need to run the planned agent once before we consider it. Let's add that now. So how we'll determine this is we'll look at the state and get the replan_flag. If the replan_flag is true, then we'll discover the planned_agent and get that agent, and we'll run that planned_agent by routing in the goto. We'll also iterate the current_step and set the replan_flag to false in this case. Now that we've created the Planner and the Executor, we can go about creating our sub-agents. These sub-agents will be used by the Executor to take actions against the plan and reach the user's goal. The first sub-agent we'll define is the Web Researcher. This is going to do the bulk of our data tasks, getting information from the web, ready to generate charts and synthesize and respond to the user with an answer. Let's take a look at how we create that Web Researcher. To start building our web search agent, we're going to use a ReAct agent from LangGraph. A LangGraph React agent will use a tool along with an LLM to accomplish its task. This will be the basis of our sub agents here. The tool we'll use for the web search agent is Tavily. We'll import TavilySearch from langchain_tavily and then we'll define our Tavily tool. And we'll pass in a max_results to five. This is the number of results we'll get from the open web. To see what this does in the tool call alone, we can invoke the tool call directly, just like so. Here, we'll ask what JP Morgan's current stock price and return the results. In the results, we can see a list of URLs, titles, and the content along with some other information for each result. This is what the ReAct agent will use to answer the question. This would be a lot of information to pass back to our master multi-agent system. So by using a ReAct agent, we get to actually provide only the information that the agent system needs to answer the sub-question provided to that sub-agent. Now let's build our ReAct agent using this tool. To build our react agent here, we'll import the agent_system_prompt from the helper module and set the LLM to gpt-4o. Then, we'll create the web_search_agent by calling create_react_agent, passing in our chosen LLM and tool, the tavily_tool, and then providing a prompt. In this prompt, we tell it that it's the Researcher, and that it can only perform research using the provided search tool, tavily_tool. When it's found all of the needed information, it should end its output and not take further actions. Executing the cell will initialize our web_search_agent, and then we can try it out. Now, we can execute our web search sub-agent with the same query we did before and ask JP Morgan's current market cap. By using a web search agent armed with the tool, rather than the tool alone, we can provide only the needed information back to our agentic system. Now, we got a nice clean answer containing exactly what JP Morgan's current market cap is without any extra information that we don't need. The last thing we'll do when we define our web search agent is to wrap it in a LangGraph node. This allows us to build it into a graph. The web_research_node will take in the state just like every node. and it will also route to the executor. That way the executor can choose the next action in the plan after the web_research_node. Then, we'll retrieve the agent_query from the state. This is the exact sub question that's being passed to the web_search_agent. Then, we'll invoke our web_search_agent, just like we did before, using this messages key and passing in the agent_query. Last, we'll wrap this response in a HumanMessage. This is the type that will allow us to wrap this response in a human message type from LangChain. And we'll go back to the executor using the Command, along with updating the state. Here, we'll return Command containing an update to the state and the next action agent to take. In the update to the state, we'll update the messages which tracks our conversation history. between our different sub agents with the result from our web search agent. The resulting messages from the web search agent get appended onto the conversation history in the messages key in our state. Now we've created our web search agent. Next, we'll go to the chart node. All right, we just finished creating our web search agent. This is going to do our data tasks for this agent system. Now, we can create our charting agent, the next sub-agent in our architecture. Our chart_agent, just like the web search agent, will be a react style agent. This means we're binding the LLM to a tool. But in this case, rather than a web search tool, we'll use a python_repl_tool. This is going to allow the LLM to execute Python. Particularly, it's going to allow the LLM to execute Python. in order to generate a chart. We'll give it the prompt that it's tasked with generating charts and working with a researcher colleague. We tell it to print the chart first and to save it to our current working directory, and then to tell the chart summarizer exactly where the path is and some notes summarizing the main insights. We'll also wrap this agent in a LangGraph node. This agent again is going to take in the state, tracking all of these same information in the agent's memory. and it's going to route into the chart summarizer. The first thing we'll do is we'll invoke the chart agent using the state. and then we'll take out the content and wrap it in a HumanMessage. These messages are going to be passed into the state and appended to our conversation history. tracked between the sub agents. As we mentioned in the type hint, we'll also go to the chart summarizer using the command. Following from the chart generation agent, we'll create our chart summary agent. The chart summary agent will take the image, the chart itself, and provide a nice textual response back to the user. Similar to the last of agents, we'll also create a react agent here. But this time, we won't use any tools. We'll also provide a prompt that tells it to generate image captions, and its task with generating a standalone, concise summary of the provided chart saved at a local path. Then we can wrap our react agent in a chart_summary_node. The chart_summary_node is going to invoke our react agent and then pass to the end. The next agent we'll create is our synthesizer agent. This synthesizer agent is going to directly invoke our LLM with a provided prompt so that we can take in the entire information that the agent has already gathered. and provide a response back to the user. For our synthesizer, we'll again use gpt-4o as our LLM. And then we can create our synthesizer node. Our synthesizer node is going to gather the informative messages that it needs for the final synthesis. We'll do that by extracting the message history from the state and then finding any messages from the important agents, the web researcher, the chart generator, the chart summarizer. Notice that this doesn't include the planner or the executor. Then, we'll fetch the user_question from the state as well. That way, the Synthesizer is armed with the exact user query it needs to answer. Once we have this information, then we can create our synthesis_instructions. This is the exact prompt we'll pass to the LLM to tell it how to answer the question. This is very stylistic. We tell it to perform any lightweight comparisons or inferences required, but not to invent any facts not supported by the context. We'll tell it to provide a concise response that fully answers the question and to start with a direct answer and including any citations if it needs, and to keep the output crisp. Now that we have our relevant messages, the user question, and the synthesis instructions, we can build our prompt containing each of these key pieces of information. Our summary_prompt is going to be a HumanMessage with the content of the user_question followed by the synthesis_instructions and then last, followed up with the content from all of the relevant_msgs in that conversation history that we've been tracking in the state. Once we've built our prompt, we'll invoke our LLM. with this prompt exactly. When we receive the LLM reply, we'll extract out the content and then we'll print the Synthesizer answer, just so we can see it. And we'll finish off by routing to the end and updating the state with the final answer and our last set of messages. Now that we've created each sub agent in our graph, we can start to build the agent graph. Let's remind ourselves of the architecture we've been building. So now, we've created all of the nodes we need to finally build out this workflow. We started with the Planner and then the Executor, and then we built out each of the sub-agents needed to accomplish goals according to the plan. We created the Web Researcher, the Chart Generator, the Chart Summarizer, and the Synthesizer. Now, we can build the graph. To do that, we're going to import StateGraph from langgraph. And we're going to instantiate it with our custom state. Then, we'll add each of the nodes to our workflow that we've been creating in this lesson. Last, we'll add a single edge from the START to the planner. so that we always start with a plan. Now, we can compile our workflow into the variable named graph. To verify that we've created the graph we intend, we can draw a mermaid chart from this graph using a provided LangChain utility. This gives us exactly the architecture we intended. It looks a lot like the slide we saw before, doesn't it? Now that we've created our graph, we're ready to go and ready to start using our agent. So the first question we'll ask the agent is to chart the current market capitalization of the top five banks in the US. We expect this to rely on web search to capture the current market cap of these banks, and then go to the chart node to generate a chart, and then last, the chart summarizer to provide a textual companion response. We'll invoke the graph with the state, including the user query. We'll start off our messages history with that user query right at the beginning. And then we'll also provide a list of enabled agents. These enabled agents are slotted into the planning prompt, so that the planner only writes a plan with the available agents, and the executor prompt, so that the executor only chooses agents available and ready to use. So our agent has responded with an answer. It's provided us a nice chart with the top five market caps of US banks with JP Morgan in the lead. The chart summarizer, however, did not do as good of a job. It just provided us with the CHART_PATH, but didn't actually give the textual response we're looking for. In other cases, the LLM will actually give the full textual response, and you might see that in your notebook. But either way, we'll observe exactly how well this agent is doing when we get into evaluations later. Let's try a second query. In this query, we asked to identify regulatory changes for the financial services industry in the US. Compared to the last example where we were looking to generate a chart and the chart summarizer. Here, we're taking the synthesizer route. We're looking to only respond with a text response. Just like before, we're going to invoke the graph with our user query and initialize messages with that query, and then the list of enabled agents that we have to work with. Let's wait for the agent to return a result for us. So our agent has responded with a nice detailed answer to answer our question. It gives us key details about regulatory changes in Congress, the SEC, the Federal Reserve, and more. And it even provides citations about where it got this information from. In the next lesson, we'll go on to expand our agent's capabilities and allow it to reason over internal data as well, using text-to-SQL and document search. These will be additional sub-agents provided into our system so that we can reason and answer even harder questions and provide more value to our end user. Let's get to the next one.
