Sign in to continue learning
Evaluating AI Agents
Beginner
2h36m
Topics
Agents
Evaluation and Monitoring
GenAI Applications
LLMOps
MLOps
Prompt Engineering
Task Automation
Now that you understand your agent example, the next step is to add observability to your agent code. This will give you insight into the trajectory or sequence of steps your agent takes to respond to a user's query. You'll learn how to instrument your agent, which means adding code to track your agent steps, and then visualize the collected information or traces in the Phoenix UI. All right. Let's have some fun. When it comes to observing your agent, there are a few key concepts that are important to know. The first is observability. Observability is a general software concept that refers to having complete visibility into every layer of your application. In the context of LLM applications, that's often tracking things like the prompt, the response, token usage, and then any calls that happen around the calls that you're making to LLMs. Observability is typically made up of traces and spans. Traces refer to full run through of your application, so a single trace would be one and end run of your application from an input all the way through to an output. And then traces are comprised of multiple spans. Spans are individual steps of calls to LLMs or code, or lookups to databases or whatever might be. So one trace is made of multiple spans, and they're typically presented in this hierarchical way where spans can be nested underneath each other to show that they're operating within another span. And then a set of spans will make up one trace hierarchy. Spans are typically presented in this hierarchical manner here where you have individual spans that are nested within each other, all making up one trace. And you might have spans for LLM calls as you see in the image here. You might have spans for tools. You might have chain spans which are just general logic steps that happen within a trace. Traces and spans come from a framework called OpenTelemetry. OpenTelemetry is often abbreviated to OTEL, and it's one of the most widely used standards for application observability, even outside of LLMs and AI agents. OTEL includes the idea of traces and spans that are captured within your application, as well as some standards for how to capture those traces and spans. Often this process is referred to as instrumentation, which you'll learn more about in a second. And then OpenTelemetry also includes the concept of collectors and processors to receive those traces and spans, and then enumerate them in a platform that allows you to visualize those traces and run later evaluation on them. Throughout this course, you're going to use a tool called Arize Phoenix, which serves as one of those collectors to allow you to receive, visualize, and evaluate traces. This is what a trace looks like in Phoenix. You're going to see a lot more of this throughout the notebooks as you start implementing some of these techniques. But you'll notice there's one trace here of Agent Run in the image that's made up of multiple spans. You have an LLM span in orange. You have your chain spans in blue. And then your tool spans in yellow. Traces and spans are captured in your application through a process called instrumentation. Instrumentation refers to the practice of marking which functions or code blocks within your code you want to track as spans and what attributes you want to attach to those spans. You can do this manually with clauses or decorators, as you'll see in the next notebook as well. However, tools like Phoenix do automate some of this process for you, especially if you're using popular libraries like OpenAI or LlamaIndex or LangChain. Observability is important for a few different reasons. First, when you're getting off the ground and building your application, observability really simplifies the debugging process. As you're building. It's much easier to debug by going through a nice visual trace than it is to pore over print statements and logs in your application, so it makes it easier to get going. But then, as you start launching your application to multiple different users, you start moving towards production or testing even. Then, it gives you a detailed log of all of the calls made to your application and all of the different inputs that you're receiving. So it gives you a large database of information of all of your different application runs, so you can monitor how it's performing. And then these traces will become the bedrock that is used to run evaluation. So later notebooks will have you export data from Phoenix that you can then use for evaluations at scale across multiple runs of your application. Finally, putting these together, observability helps you really start to understand and eventually control some of the more unpredictable behavior of LLMs. LLMs are by their very nature, unpredictable in some cases, and the best way to deal with that is by monitoring them and later evaluating them within your application. In this lesson, you've learned what observability is, how it works, and why it's important, as well as some of the basic building blocks of observability like traces and spans, and how those traces in spans are collected. In the next notebook, you'll implement some of the instrumentation techniques that you saw in these slides, and you'll set up your first Phoenix instance here to collect those traces.
