Welcome to Building and Evaluating Data Agents, built in partnership with Snowflake. In this course, you'll implement a data agent that can perform web searches and connect to data sources to collect data, analyze it, and visualize results and provide insights, even more importantly, you apply offline evaluation or offline evals to improve and iterate on your agent's design, as well as real-time, also called inline evals. that the agent can use to adjust its strategy even while it's running. I'm delighted that the instructors for this course are Anupam Datta, who is AI research lead, as well as Josh Reini, who is developer advocate at Snowflake. Anupam was formerly a professor at Carnegie Mellon University and founder of AI observability startup Truera, that was acquired by Snowflake. Josh also previously worked at Truera and continues to build and maintain TruLens, which is an open source library for agent tracing and evaluation. Thanks Andrew. We are excited to work with you on this course. Building evals is a really important skill for building agentic workflows. This course will show you some important best practices. In fact, I learned quite a bit from Anupam's last short course with us and I think you'll learn a lot from this one as well. And specifically, you start by assessing your agent's answer quality. So you use an LLM as a judge to measure the end-to-end performance of the agent and evaluate that the final answer is relevant to the user's query, as well as whether the data retrieved for the analysis is relevant to the user's query. And finally, if the final answer is grounded in the list of all the retrieved data. You can use these offline evaluation metrics to improve your agent's logic or update its prompts or maybe try out different LLM models. Additionally, you also use LLM as judge to evaluate the process by which the agent used to arrive at that final answer. This technique can also interestingly, be used not offline, but instead at runtime to give feedback to the agent even as it's running, to help the agent adjust its data retrieval and analysis strategy. You'll implement your agent using a multi-agent workflow that consists of a planner. a plan executor and specialized agents such as web and data researchers, a data visualizer and response synthesizers. The planner takes the user's query and generates a plan. And the plan executor directs the plan's execution and determines which specialized agent should proceed based on the plan. Each agent shares the output of its execution step and the plan executor through the agent state. After each retrieval step, the plan executor reflects on the plan and decides whether to update it. You will evaluate both the quality of retrievals and the quality of the plan. Was the plan aligned with the user's request? Were the agent's actions aligned with the plan and executed efficiently and logically? You will then improve this agent by adjusting its prompts and implementing in-line or runtime evaluations. The inline evaluations will provide the plan executor with real-time scores for context relevance immediately after any retrieval step. This will inform the plan executor whether the retrieval step lacked context relevance. And if so, the plan executor will ask the planner to adjust. After you implement these improvements to the agent's design, You will compare the offline evaluation metrics and both offline and real-time evaluations rely on tracing the agent that you will also learn about in this course. Many people have worked to create this course. I'd like to thank from Snowflake, Allison Jia, David Kurokawa, Daniel Huang, Nikhil Vytla, and Shayak Sen. From DeepLearning.AI, Hawraa Salami also contributed to this course. In the next video, you'll learn about the three dimensions of evaluation for an agent. It's goal, plan, and action, or GPA. Let's go on to the next video.