Sign in to continue learning
Nvidia’s NeMo Agent Toolkit: Making Agents Reliable
Intermediate
Topics
Agents
It's time to dive into NAT evaluation's framework to measure agent and workflow performance. You'll create datasets with ground truth, uncover bugs, and use insights to make data-driven improvements with confidence. Let's get to it. In this lesson, we're going to learn about agentic evaluation. Evaluation is super important for agents. Evaluation can often be overlooked in a complex agentic environment. NAT makes it straightforward to run evaluations against your workflow. So in this lesson, we will create evaluation data sets of question and answer pairs, and we'll use that data set to test if our agent is actually giving us a correct answer. And along the way, we will discover a bug in a production workflow and we'll fix it. Why would you evaluate your agent? Often times, we'll ship something into production and it'll work. But something can drift and we might not catch it. Maybe we update a model or maybe we change a prompt. Evaluations help keep our agents honest. NAT has several evaluation frameworks built in. Of course, it's pluggable, so you can add your own evaluation framework if you have one that you like. But it's just a simple matter of adding evaluation properties to your configuration YAML and running NAT Eval. So what is NAT's evaluation flow? You give it a dataset with a list of inputs and expected outputs. and NAT will run those evals against your agentic system and it will score how many were correct and how many were wrong. Of course, you might have your own evaluation framework that can't be expressed using input and output pairs. NAT's eval system is pluggable, so you can just create a wrapper around your own evaluation framework. What does this look like in config? So we have a new top-level property for our configuration file called eval. And this eval will take a few other properties. The general property is where we tell it where to dump its output and where to load the datasets from. And you can configure one or more evaluators. In our case, we have one evaluator and we've named it answer_accuracy. And it is a ragas evaluator, which is an evaluation framework you might be familiar with. And it's built into NAT. Ragas has many different metrics that you can pick from. In this case, we've picked AnswerAccuracy. You can see the other metrics in our documentation. and we've handed it an LLM. You can see our question and answer pair here, the average temperature in Austria. And we pass that into the simple eval JSON dataset. and then we call nat eval. And this is the power of Config-Driven Development. Instead of changing code, rewriting a lot of Python, hoping we don't break it. We can do experiments in our config. We can change functions. We can swap out LLMs. If we have our eval set up correctly, we can make these changes and run our evals and be confident that our changes are correct. Of course, you can add more than one evaluation to your config, as many as you like. And you can use them in tandem with other features that we've already looked at, such as observability. You could run your evaluations in parallel while they're hooked up to Phoenix, and you can watch the trace data come in as the evals run. Without observability, we don't know how our agent arrived at a correct answer. Without evaluations, we don't know if our agent arrived at a correct answer. With NAT's evaluation framework, we can score multiple evaluation question and answer pairs and be confident that our agent is ready to go. In this notebook, we are going to look at evaluation. How do we find and fix bugs with NeMo Agent Toolkit? Let's go ahead and install the climate_analyzer package that we've been working with and run some evals against it. Now that that's installed, let's look at our evaluation dataset. So we have a very small evaluation dataset here. It contains one question and one answer. What was the average temperature in Austria in 1980? and the average temperature in Austria in 1980 was 6.8 degrees Celsius. Let's find what our climate data says about Austria in 1980. We get some values back here. Let's calculate the average to confirm our ground truth. And we see from our climate data that the average temperature for Austria in 1980 was 6.8 degrees Celsius. How do we run evaluations in NAT? Let's look at a pre-written evaluation config file to see. This file should be familiar to you. We have our LLMs that we wrote in previous notebooks with our climate_llm and our calculator_llm. We have the functions that we've been working with, climate functions and calculator functions. and we have our workflow. But now we have added an evaluation config section. Eval is a top-level section, and within eval we can add several evaluations sets. First, we add a general property here that tells us where to store our evaluation data, whether to clean up that evaluation data after a run and where our datasets come from. And now we can list our evaluators. There are many different types of evaluations you can run. Maybe you have some you've written yourself, maybe you're using standard evaluations such as Ragas. And that's what we're going to use here. So we'll name this answer accuracy. and the type is ragas which is built into NeMo Agent Toolkit. Ragas needs a metric. There are several different metrics such as groundedness. In our case, we're going to pick one called AnswerAccuracy. And you can find these metrics in our documentation. How does it evaluate this metric? It needs an LLM and so we will go ahead and pass it the pre-configured climate_llm. And that's going to check our accuracy. We've seen NAT run and NAT serve, and now we're going to see a new command, nat eval. This takes the config file that we looked at earlier, and it runs the evaluations in there against the workflow defined. The evaluator is going to write files out to the location that we put in the config. And so we see here that it's written an answer accuracy output JSON file. Let's see what's inside that file. So we can see the evaluation results. The average score was zero over one, which means that we evaluated very poorly. So what was the average temperature in Austria in 1980? This was the question. We expected to get 6.8 degrees Celsius, but instead we got 9.574 degrees Celsius. Therefore, we missed our only evaluation question, so we got zero out of one correct. Now let's look at the reasoning steps that it took in order to get to that wrong answer. So this is a lot of code. The evaluator puts out quite large and verbose JSON files that will help you track down what went right or wrong in the evaluation. So this code just parses that JSON file so that we can see the results a little bit more cleanly. So we can see the reasoning steps here of the evaluation. It tried to pull the average temperature in 1980, but it didn't even pass the year to our calculate_statistics function. And because of that, it got an average for the entire climate period 1950 to 2025. However, it gives us a very confident answer back. And so, if we didn't run evals, we probably would miss this. It's not obviously wrong and the agent is confident in its answer. But since we have grounded evals that we can now run against our agent using NAT, we can see that it's incorrect. So let's run this with a config that more directly tells the agent to call with the correct year. Now that that's run, let's load these evaluation results and see that they are correct. Now we can see from our evaluation results for the one question answer pair we had, that we have one correct answer out of one question. And we correctly got 6.8 degrees Celsius. So in this notebook, we've seen how to build an evaluation dataset, how to run it against our agentic workflow using NeMo Agent Toolkit. We've looked at a real world use case where the agent confidently gave the wrong answer and we were able to discover that through running evals. Then we were able to fix that bug and push our agent right back out into production.
