In this module, you've learned how to build a benchmark. You started by defining requirements: clear statements of what the application should and should not do. Then you turned those requirements into test cases with the information you gave the application and the outcome you expected. You ran those cases, reviewed the results, and labeled the failures you found. And finally, you looked for gaps in coverage, added useful cases, and curated the set so each case had a reason to stay. Now let's apply that full workflow to Northwind. The goal is to build the benchmark you'll use in the next module when you start automating those judgments with evaluators. Start with a behavior you already found in a trace. A customer asks, "How should I care for my merino wool sweater?" The agent looks at product information, which says machine washable on wool cycle, but the agent answers, "You can machine wash it on wool cycle and then tumble dry." The first part is supported by the data, but the second part is not. The agent had information about washing, but it didn't have any information about drying. And that unsupported detail matters, because care instructions can affect what the customer actually does with the product. So, write down what actually went wrong. The agent recommended tumble-drying, even though Northwind didn't provide drying instructions for this sweater. Next, turn that observation into a requirement. You could write, "Do not recommend tumble-drying the merino wool sweater." But that's too narrow. It only covers this one product and this one unsupported claim. The same kind of problem could happen with sizing, color, availability, materials, or return policies. So make the requirement broader. Only make product claims that are supported by Northwind data. Now you have a standard you can test. When the agent answers a product question, you can ask: Does Northwind data support the claims in this response? Next, turn that requirement into test cases. Just the sweater case is not enough. You want a small set of cases that checks the requirement under different conditions. One case should have complete information. For example, a customer asks, "What colors does the Classic Tee come in?" And the catalog data lists the available colors: white, black, navy, and heather gray. Here, the expected outcome is straightforward. The agent should answer with the listed colors, and avoid adding colors that are not in the catalog. But what if the agent only has partial information? A merino sweater case fits here. The customer asks how to care for the sweater. Northwind data says it's machine washable on a wool cycle, but gives no drying instructions. So the expected outcome is: use the supported washing instruction and do not add drying advice. The agent can say the drying instructions are not available. A third case should have no relevant information Suppose a customer asks, "What's your return policy?" but Northwind does not provide any return policy information to this agent. The agent is expected to say that the information is not available or that it cannot help with that request. Now you have three cases that test the same requirement from different angles: complete information, partial information, and missing information. You can also vary the customer side. For example, "What colors does the Classic Tee come in?" and "Do you have the Classic Tee in size medium and what colors are available?" are two distinct requests. The second one asks about size and color, so it may require the agent to handle multiple product details at once. But if two cases only change the wording without changing what the agent needs to decide, you may not need both. The point is not to collect as many phrasings as possible, but to keep cases that test meaningful differences. Once you have the cases, run the agent on them and review the results. For each run, compare the actual versus the expected outcome. If the customer asked about colors, and the agent listed only colors from the catalog, that run passes. If the agent adds forest green when the catalog only lists white, black, navy, and heather gray, that run fails. Don't forget to write down why. In this case, the agent mentioned a product color that was not supported by Northwind data, then label the failure. That could be a groundedness failure, and more specifically, hallucination, because the agent made a claim with no supporting evidence. Now look at the return-policy case, where the customer asks, "What's your return policy?" Northwind has no return-policy information, so if the agent says, "I do not have return Policy information available", that meets the expected outcome. But if it says, "You can return items within 30 days," that fails. Here, the main problem is that the agent answered a question outside the information Northwind can provide. So you might label that as a scope violation. The expected outcome tells you what should happen, while the failure label tells you what went wrong when the agent did something else. Keep both with the case, As you review more cases, start looking across the set. Do you have several unsupported product claims? Does the agent ever invent policies outside Northwind's scope? Do you see cases where the agent should say information is missing? Are there cases where the agent should answer directly because the data fully supports the answer? These questions help you find coverage gaps. Suppose your first cases mostly test care instructions. That's useful, but the requirement is broader than care instructions. You may want cases about colors, materials, sizing, availability, or product compatibility. Suppose you have several cases where the information is missing, but very few where the information is present and the agent should answer directly. Then your benchmark may overrepresent refusals and underrepresent ordinary successful answers. You need both. A useful benchmark should catch failures, but it should also confirm that the agent still handles normal cases correctly. Next, add cases to fill the gaps. Start with real examples when you can, which you can find in traces, reviewed runs, user questions, bug reports, or support examples. If you find a real customer question that tests an important gap, use it. Real examples help the benchmark reflect situations the application actually encounters. Just bear in mind that real examples may not cover every case you need. If you need a specific kind of case and cannot find one, you can create it yourself, or use an LLM to generate a candidate. For example, you might ask for a customer question where the catalog contains some product information, but not enough to answer every part of the request. Then review the generated case carefully. Does it test the situation you meant to test? Is the product data realistic? Is the expected outcome clear? Did the generated case quietly change the requirement? only keep the case if it adds useful coverage. As the set grows, you should curate it. Some cases will be redundant. For instance, "Is the Classic Tee available in medium?" and "Do you have the Classic Tee in size M?" may test nearly the same thing. you probably don't need both unless the wording difference matters. But a case that asks, "Is the Classic Tee available in medium, and what is your return policy?" does add a different challenge because the agent needs to answer the product question using catalog data, while also avoiding an unsupported return policy answer. So that case may be worth keeping. A useful case should do at least one of three things. It should either cover a meaningful situation, protect against an important regression, or clarify a boundary in the requirement. If a case does none of those, remove it or revise it. Also pay attention to difficult cases If your benchmark has 98 straightforward product questions and only two difficult missing information cases, the overall score can look strong even if the agent fails both difficult cases. That doesn't mean the benchmark is wrong. It means you need to read the score in context. The overall score describes performance on the benchmark you built, not the true frequency of success or failure in production. During development, it's often useful to include more difficult cases than you expect to see in everyday use. That gives you enough examples to measure whether the agent improves on the behaviors you care about. At the end of this process, each Northwind benchmark case should keep together the pieces you need later: the customer query, the relevant product data, the expected outcome, the source of the case, and when you've reviewed a failed run, the failure label that describes what went wrong. This is what a complete case might look like. customer query, "What is your return policy?" Northwind data has no return policy information available. The expected outcome would be, say that the information is not available. If the agent invents a return policy, label the failure as a Scope violation. Another case might be: Customer query, "How should I care for the Merino Wool Sweater?" Northwind data says, "Machine washable on wool cycle." The expected outcome is: Use only the supported machine washing instruction, and do not add drying advice. If the agent adds unsupported drying instructions, label the failure as Faithfulness drift. And the third case might be customer query, "What colors does the Classic Tee come in?" Catalog data indicates "White, black, navy, and heather gray." The expected outcome is "Answer with the listed colors." If the agent mentions another color, label the failure as a Hallucination. Once the cases are reviewed and curated, save them as a Northwind benchmark. That benchmark now contains requirements, test cases, expected outcomes, reviewed judgments, and failure labels. You can run it again as the application changes, and because the cases stay the same, you can compare versions against a stable standard. Keep that in mind for the next module, where you'll go into the evaluate stage of the evals process. You'll build evaluators that check whether Northwind responses meet these requirements, and you'll test whether those evaluators agree with the reviewed labels you created here. See you in module three.
