Welcome to Building and Evaluating Data agents built in partnership with snowflake. In this course, you implement a data agent to perform web searches and collect the data sources. So collect data, analyze it and visualize results and provide insights. Even more importantly, you apply offline evaluations offline evals to improve and iterate on your agents design as well as real time. Also called inline evals that the agent can use. Adjust its strategy even while it's running. I'm delighted the instructors for this course are only about data with AI research lead, as was Josh Brady, who is developer advocate at Snowflake and was formerly a professor at Carnegie Mellon University and founder of AI observability startup True ERA. Those acquired by snowflake. Josh also previously worked at Torreira and continues to build and maintain True Lens, which is an open source library for agent tracing and evaluation. Thanks, Andrew. We are excited to work with you on this course. Building evals is a really important skill for building HNC workflows. This course will show you some important best practices. In fact, I learned quite a bit from Anupama natural close of us, and I think you learn a lot from this one as well. And specifically you start by assessing your agent's answer quality. So you use an error message to measure the end to end performance of the agent, and evaluate that the final answer is relevant to the user's query, as well as whether the data retrieved for the analysis is relevant to the user's query. And finally, if the final answer is rounded in the list of all the retrieved data, you can use these offline evaluation metrics to improve your agent's logic or update its prompts, or maybe try out different models. Additionally, you also use elements Judge to evaluate the process by which the agent use arrive at that final answer. This technique can also increasingly be used not offline, but instead at runtime to give feedback to the agent, even as is running to help the agent. The Justice data Retrieval and Analysis strategy. will implement your agent using a multi-agent workflow that consists of a planner, a plan, executer, and specialized agents such as web and data researchers, a data visualizer, and response synthesizers. The planner takes the user's query and generates a plan, and the plan Executer directs the plan's execution and determines which specialized agent should proceed based on the plan. Each agent shares the output of its execution step and the plan executed through the agent state. After each retrieval step. The plan execute A reflects on the plan and decides whether to update it. You will evaluate both the quality of retrievals and the quality of the plan. Was the plan aligned with the user's request? The agent's actions aligned with the plan and executed efficiently and logically. You will then improve this agent by adjusting its problems and implementing in line or runtime evaluations. The inline evaluations will provide the plan executer with real time scores for context relevance immediately after any retrieval step. This will inform the plan Executer whether the retrieval step lacked context relevance, and if so, the plan Executer will ask the planner to adjust after you implement these improvements to the agent's design, you will compare the offline evaluation metrics, and both offline and real time evaluations rely on tracing the agent that you will also learn about. In this course. Many people have worked to create this course. I'd like to thank from snowflake, Allison Chia, David Kurokawa, Daniel Huang, Nicole Victoria and Jackson from deep in the eye are out. Salameh also contributed to this cause in the next video, you learn about the three dimensions of evaluation for an agent is go, plan and action or GPA. Let's go on to the next video.
Building and Evaluating Data Agents