Log In to continue learning
Evaluating and Improving Agentic Applications
Intermediate
15h7m
0
Hey, Aparna, welcome to the course. Why don't you introduce yourself to the audience and let everybody know who you are? Awesome. Hey, everyone, my name is Aparna. I'm one of the founders of Arize. have been in the evals and observability category for many years now, and it's a deeply exciting category to be a part of. Cool. So one of the challenges we had putting this course together is that coding agents are advancing so quickly. A lot of stuff that used to be manual effort is now something that you'd ask an agent to do. what's not in that category? Where is human judgment indispensable? That's a great question. I mean, on one hand, I think that if you're still doing things the way that you were doing them last year, it's probably not the right thing. Um, but I also think that there's, you know, human judgment and being able to discern what is really good and bad for your business, for your use case, is still something that we really truly need domain expertise to do. So I think some of the best things that we've seen AI engineers or AI PMs do today is take that judgment and use that to define what really good evals or what data even needs to be traced and observed in order to write those evals. Cool. So, a related question to that is, you've said in the past that agents are now capable of coming up with the evals themselves. Uh, but we also say that, as you just said, domain experts are a huge part of evals. Let's, let's square those up. Where do you think that an agent can help, uh, put an eval together, and where's the line where the domain expert needs to get involved? Yeah, great, great question. I mean, So first and foremost, I think, you know, there was a lot of, there's just a lot of traces and telemetry data that is being emitted right now. You have agents that are all being traced. You have agents that now review those traces. You probably have agents that are reviewing those agents on top of it. And so the volume of telemetry that is being, you know, just collected right now is massive. And I just don't think, you know, observability 5 years ago, 10 years ago used to be humans looking at dashboards and looking at metrics and logs. There's just the volume of data has grown to a point where, and maybe was always, you know, such a great use case for agents, is that you're going to have agents running through those traces. and what they're gonna be really good at doing is figuring out patterns of what are types of questions that your users are asking? What are questions or common types of errors that are occurring for your user experience? And what agents can really do is identify categories of things that might be even beyond the categories of what you might have thought evals should contain. So they're they're probably really good at figuring out the unknown unknowns. And so what I think is a really great balance between the two right now is one, you know, as the human judgment is really great at figuring out what do you want the end-user experience to be for the product or the agent that you're actually building, what do you want it to be doing? And so there's probably a core set of do you want them to leave feeling less frustrated? Do you want them to kind of reach an outcome um of the use case that they're building towards. And then, you know, that's a great way to get an initial set of evals going. What you can also do with these agents crawling your traces is figure out what are a whole other, you know, types of behaviors and signals that you can gather from the traces to figure out new types of evals. And then how do you continuously improve those evals so it's not just a, you know, one-shot approach? You're kind of continuously refining that based off of human judgment and the new traces that you're getting is kind of a great way to kind of use them both, the human judgment and the agents, to build better evals. I think one of the things that's also changed a lot You probably agree with this, too, is even two years ago, we were spending a lot of time figuring out the right way to prompt things. It probably doesn't matter anymore, and so you can probably have the agent write a way better prompt for the evals, way better, define a way better task for other agents than you probably will at this point. And so, it's crazy to think, but really it is comes down to taste. Can you... define what good and bad is, and help the agent learn that from your judgment. One of the ways that I've seen it expressed is that humans now have to operate at a higher level, a higher level of abstraction on top of a loop that you've built for your evals. Would you say that's a fair summary? I think that's a great way of putting it. It's, you know, at first engineers were writing very, very low-level code, and then there was abstraction languages on top of that. And now, you know, engineers don't even code. They write prompts, and now it's like you're not even prompting individual eval templates. You're probably prompting how do you have these agents that keep those eval templates up to date. And so there's levels to this. So, one thing I've heard you talk a lot about is the coming of agents as judges instead of LLMs as judges. How is an agent as a judge different from an LLM as a judge? Why do we need agents as judges? That's a Good question. Um, so I think what we realized was that, you know, we had spent years of individuals who were building, let's say, slightly less complex applications or agents, where this is kind of a decade of more and more autonomy in the agents that are being built. Um, and so LLM-as-judges I think And to be honest, they're still really useful today as well, but they're really great for evaluating the quality of the outputs. But where they tend to have gaps is understanding the entire journey that it took or the entire trajectory that it often took in order to debug an application. and, ultimately, how do you then suggest improvements or fixes? And so agent as a judge is, think about it as instead of a single prompt evaluating the outputs of these agents, it is an entire agent evaluating the outputs of another agent. And so it can adapt criteria. It can modify how deep it needs to go in order to get more information to properly evaluate the quality of the output. And then it can be, of course, kicked off. It can kick off either other agents or kick itself off to go and put up improvements to fix. the issues that it had detected. And so agent as a judge feels like an evolution of having these judges become more autonomous in the way that they evaluate other agents, and kind of giving them a way to also close the loop and start improving the agents. What are some of the failure modes for agentic judges that somebody putting them together can be cautious of? Because, obviously, you're talking about the frontier of evals, and not everything's going to go perfectly. I mean, I don't know if you think about this this way, Laurie, but sometimes there's just like a slop loop that ends up getting created where there's one piece of slop, and then it just kind of snowballs. And it's the one thing that I think about where human intervention and human judgment ends up becoming really important, so that the evals and how you're thinking about quality kind of stay You know, they have to evolve. I don't think it's a one-shot type of process, but I think they kind of have to you need to give it nudges so that it doesn't go you know, down and think something's really an issue when it quite might not be. Mhm. And so that nuance is sometimes really helpful, and I think that that's it just so much, again, depends on the application. You know, we obviously have an agent built inside of our own product, Alex. And I think we've learned a lot from, you know, kind of dogfooding and writing evals for our own agent. And sometimes, you know, we realize that it can take Alex you know, a couple different turns to understand ultimately what the user wants. But if the user's question didn't often start asking that, and actually their true intent was something else, sometimes the turns that they took was actually fine, probably would have been similar to what a human did. But if it took those turns even when the user was super direct, then then that's an issue. And so, having that ability to figure out when you know, the trajectory is valid, and when it's not, I think is is really interesting. Um, I think we're also going to see a ton of not just quality, you know, there's a lot of quality, um, evals that teams tend to write on the actual outputs of the agents, but I think there's deeply more and more, think about it more like cost type evals or safety type evals to kind of consider is the output worth the ROI that was invested? Speaking of agents inside of our products, uh Arize actually has Signal, a built-in agent that reads your traces for you and suggests fixes to your app, which I think we can agree is pretty cool. How do you see that fitting into the observe, benchmark, evaluate, improve workflow that we've been talking about in this course so far? I mean, I think it really is kind of just the, future of how we're going to build these loops. In the beginning, there was traces that were collected, and, you know, humans had to go through, read those traces, and look and try to find where the errors are in those traces. Well, now you can have agents go through, and again, it doesn't replace the human, but it does a first pass to identify categories of issues and kind of—hence why we call it signal— find the signals within the traces. And, you know, I know so many of us talk about this, but the most important way to improve your agent, I really think, is actually reading your traces and understanding the telemetry. Even just spending and reading through five, six, seven traces, you start to understand. I feel like I end up walking away learning something from our own agent's traces. So, you know, Signal first kind of does a bit of that heavy work and actually reviews the traces of your agent. And there's a ton of flexibility to set it up so if you wanted to sample if you want it to run on a cadence, if you want it to run on some kind of trigger. You can essentially define agents to do the agentic review of the traces. It can surface up patterns of behavior and then highlight the ones that need a human's attention. Um, it can also, if given access to a repo, if given access to additional context, it can actually go ahead and put up a fix for the issue. If there's enough evidence and it feels confident about, hey, there's a root cause here that four or five, maybe 20 different traces are kind of all alluding to, it can go ahead and put up a fix. which is pretty incredible because now, the same traces that a human had to go through, understand the pattern, and figure out what to go fix, you can kind of get the fix. You can then just test the fix, which also, of course, an agent can do, so test it, run the benchmark on it, make sure it didn't break anything, And so that that's a bit of how we're thinking about this next gen of observability is that it's no longer dependent on the human having to be the person who kicks off the loop every time. It can be an agent that kicks off the loop, and it can be a human that actually reviews, makes it better, and like you always say, moves a level up where you are now kind of the the controller or the the augmentor of that loop and so you can make it better when it puts up PRs or are helpful, and we've seen this. Like, it doesn't always get it right. How do you give it feedback so that the agents get better? So now we're going to go into a little bit of a deep cut. You've written that picking the best model doesn't fix a failing agent, and you've split the system into model, context, harness layers. Walk me through that mental model. Why is that important? What do you get out of splitting up the model that way? I mean, I think I read this on X somewhere, someone said, you know, "If you can't get the value with the models we have today, yeah, I think it's a you problem, not the model problem." And, and I think what the person was maybe trying to say there was that, even though the models are really powerful, how you actually use them, how do you build these loops, and how do you actually use the contexts, is Yeah, I still think that those are really important engineering skills. There's a whole, you know, new category around harness engineering, context engineering. Um, and so when I think about What is a loop, and you've probably mentioned this as well, is that it's how do you repeatedly, you know, improve the harness, improve the context, or I guess if you're a frontier lab, you know, improve the models? And the things that most AI engineers probably have within their control is the harness and the context. And if you're using an out-of-the-box kind of harness, like a Claude Code or Codex or Cursor harness, then really it's about how do you wield the context that you're giving the harness in order to get the best output. Yeah, I think it's just really helpful to think about it in terms of what you can control and what you can kind of change in order to test to see if you can get better outcomes. We, I think, have gone through, you know, for our own agent in our product Alex, we've actually built a harness. And some of the things we learned in building the harness around you know, giving the agent and the model access to a file system and CLI tooling and having it have access to that to do the work was just, it was, I think, some of the same decisions that some of the harnesses by the frontier labs had also gotten to. But every time we did that, we also, I think we tried to keep other things constant. Tried to keep the model constant, tried to keep the context constant, and then we tested changes to the harness to be able to evaluate and see if it really moved the needle. it's helpful so that you kind of can run tests on what changes the game for your, your agents. Mhm. Speaking of which, lots of people are concerned about being locked into a particular harness or a specific model. Talk me through how evals help avoid that problem. Well, if you know what the performance with a frontier model and, you know, an out-of-the-box harness can be, it gives you a really good starting place to benchmark "this is how good your agent is," and then is the frontier models can be really expensive. So you can swap them out for open-source models or just lower-cost models and then evaluate to make sure that the things that really matter, you're not missing out on in the quality of the the agent you're building. So, I think evals are I mean evals are kind of the best way to give yourself a benchmark and then kind of either decide to verify regressions, or it's a benchmark for you to hill climb and see where you can get better. So, in this course, we got all the way up to production. We talked about taking your production traces and bringing them into your dataset and improving your dataset that way. But one of the things that we didn't get into is post-production stuff: online evals and monitoring. What are the big things that somebody who's just getting into observability and evaluations should know about using monitoring and dashboards to be really production-ready? Yeah, I think evals ends up being a really big term, and there's kind of evals you do during development and testing so that you can again check for regressions, hill climb, but I would really equally say, evals in production are so, important to how you can detect your next set of, what are your next set of improvements, or your next set of hills to climb. Um, and so, really, the telemetry and the traces that you end up collecting, end up being a really good way for teams to run online evaluation. So, evals that run on your live production traffic and can be used to identify if there's common problems or patterns that you see from that traces, fr- from the traces. You know, online evals can have an interesting thing where they run continuously across your stream of data. evaluating your known, problems that you will tend to quantify. This is where I'm really excited about the future of agent as judges so that they can find the unknown unknowns in your traces. Exactly. Um, but I might even have a hot take here, which is that I think that the evals you run on your traces are honestly more, important for AI engineers and AI PMs, who are actually building these AI products because you don't—you know, it's just like shipping a product, you don't really know until it's out there in the real world and it's touching real users. And so you can benchmark it all day long, but until it's out there and you have real you know, real telemetry. I think you learn, you end up learning a lot more from them. Absolutely. So let's talk about the harder side of things. You've obviously seen a ton of success stories of people using evals in production. Let's talk about some of the failures. What are some of the pitfalls that people new to this space fall into that mean they don't get to value with evals? That's a That's an awesome question, actually. I think they see a lot of new folks try to write some of the most complex evals possible, so a trace eval that looks at a specific span in order to evaluate if the agent did exactly this step in its entire trajectory, and to be honest, my- my honest answer is you can start simple. can start simple, can literally be, you know, a simple is user happy, frustrated, or you know, some way for you to get some signal on the product KPI or the agent KPI, I guess, that matters, but I'd say start simple, and it might be that you actually don't want to overspecify and let the agents find as many of these unknown-unknowns in your traces. And so I think that's, maybe the thing I'd say for people who are starting new with evals. Cool. And, looking ahead a little bit, multi-agent systems are a super hot topic. How does evaluation of multi-agent systems differ from what we've been talking about in this course? Oh man. That's a great question. It's also the topic of a lot of some of the coolest research happening right now. I think two things. So, so one is, with multi-agent systems, it again comes back down to the use cases. You have kind of the product KPIs that the end user evals that I think about, I don't think those change as much, whether you used a multi-agent system in order to deliver the right outcome. And so I think the quality type evals still probably don't change that much. But I think the stuff that does matter for the people who are maybe building the harness or building the agent itself is, one, was it really more efficient to use multiple agents? Noam, I think, did a podcast on... the Dwarkesh podcast where he was saying it's not always linear. Yeah, sometimes you might need to do four agents to get a 2x, improvement in the efficiency. And so, did it actually improve? and are the agents actually, just like humans, are we collaborating effectively? And so, did they pass the information that they needed to each other? Were they able to self-organize, or did they need guidance in order to self-organize? And so those are all really interesting things I think about in more of the How did the multi-agent system ultimately arrive at the end result, or the answer, which is probably more interesting for the person who's actually building the system. to understand things like efficiency and coordination of the agents. Do you think the model capability is a barrier to multi-agent systems? Do you think we're going to see a leap forward in multi-agent systems as the models get better, or do you think it's more about the design of the multi-agent system? I mean, I think the models, I don't want to discount the models getting better at all, because I think definitely from the stuff we're reading and what the frontier labs are putting out there, I think multi-agent is on the top of everyone's mind. Um, which, you know, going back to your point, it's going to be like another layer on top. Today you are the controller of multiple agents. Well now you're going to be a director who's controlling multiple teams of agents. And so everyone's going to get everyone's going to have to move higher and higher up the stack. So I think the models, that I think will happen. The part that's still interesting to me is the harness design, because there is a lot of the merging of the harness and the model that's happened in some of the popular coding agents today. But I think there's a whole world of the open-source models and the open-source harnesses that I think we're going to see the— I'm I'm excited as the ability to really manipulate and, and kind of build on top of open source harnesses becomes more, more common. - Mm-hmm. - So for somebody who's just finished this course, what's the thing they might be tempted to do that you'd warn them away from? Like, what's the most obvious trap that they'd fall into? I'd probably go back to the thing I said before, which is building really complex evals. I think starting small, you know, evals is a way to accomplishing the product or the agent you're ultimately trying to build, and so start simple and kind of expand on it as, as it gets more complex. Cool. Looking further ahead, what's something that's definitely part of the world of observability and evals right now that you think is going to go away in the next one to two years? Oh, um, easily just the prompt testing, prompt iterations is I don't think anyone's doing that today. But I do think that there's a lot of Yeah, still education on that. Um, it's just a I think the way you prompt does not matter as much anymore. That's a really good point. So now, what's probably going to be your favorite question, is what is your spiciest take on evals? What's something you believe about the space that you think other people in the space might disagree with? I think that the decision models have kind of changed the evals category forever. Um, if you're using a frontier model to do, more decision eval, so, you know, you have a very specific yes, no classification, use case, type eval, I just don't think you should be using a frontier model for that. You're just overpaying by 10 to 100x to do that, um, for pretty much equivalent performance and significantly lower cost. So, I, I think Decision models have kind of changed the game for evals. And so now, what's interesting is, do you use a decision model to do the eval, and then you use maybe some of the more expensive frontier models to then take actions based off of that eval's decision. So that, that's where, I think the space is probably going to go. Cool, it is a super exciting time to be in evals for that reason. Thank you so much for coming and joining us on the course, and thank you for giving your perspective. Happy to be here. Thank you so much for having me.
