Sign in to continue learning
DSPy: Build and Optimize Agentic Apps
Intermediate
49 mins
Topics
AI Frameworks
Agents
Evaluation and Monitoring
GenAI Applications
LLMOps
MLOps
Prompt Engineering
RAG
Search and Retrieval
Task Automation
In this lesson, you will learn about how to use MLflow tracing to help debug your DSPy program. All right. Let's go. This lesson comes with a lab. In the Lab, you are going to build a DSPy agent with the help of an MLflow tracing. As a demo, we are building an airline customer service agent that helps users book and monitor the flights. Let's dive in. So what is tracing and why do you need tracing? Tracing basically means recording the inputs and outputs of the intermediate function inside your AI program. And the capture is a hierarchical calling stack like module A cost module B. GenAI applications can be very complex internally while only the final output is exposed. So when something is going off, it's hard to trace back to the root cause. For example, if you have built a DSPy program consisting of five submodules. Now, if one LM calls fails due to not being able to understand the prompt, even though the DSPy provides inspect history to check LM calls, it's hard to trace back. Tracing provides an easy way for interpretability and debugging. And what is MLflow? MLflow is an open-source AI ops package that streamlines your GenAI app development. MLflow helps with a full lifecycle of building GenAI applications, ensuring that its face is traceable and reproducible. Both MLFlow Server and appliance are fully open sourced. You can easily set it up. To get started by tracing a DSPy with MLflow. You only need to add a one line to a program which is MLflow dot DSPy dot autolog. Or simply MLflow dot autolog. After that, your program will be automatically traced and the trace will be saved in the MLflow server. that you can access point in time after being generated. Let's talk about what is getting traced. In MLflow which has four things inside of the DSPy program which has every module call, no matter if that's a top module or internal modules. We also trace all the cost with the adapter from which you can see how adapter formats and use query and the process the LLM response. We also trace the calls to the LM so that you can inspect the actual prompt and the actual LM response. DSPy dot tool, which is a wrapper over DSPy tool calling, is also traced. With MLflow tracing, not only we can interpret the input and output of some submodule of a DSPy program, we can also see the hierarchy of these modules. Together with some other important information like time consumption. When something is going wrong, there might be a cross mark that points out to a problematic module. The screenshot in this slide is a real trace of a react module. Let's map it to the components we traced. React is our top-level module. which calls a few some modules represented as predict, and we can see both react and the predicts are getting traced. Inside a predictor call the check adapter format and parse carries the trace for the adapter, and the LM trace in the middle has the actual prompt and a response from the LM. At the bottom, we can also see the trace of the tool calling and in this case, two according fails. So there is a red mark next to that. In a previous lesson we have seen how to use DSPy inspect history DSPy to check out the actual prompt. With MLflow tracing, now it's even easier. Simply click on the LM trace and you can see the actual prompt and LM response. Let's get started with coding. Similar to the previous lab, when you set setup the DSPy key and additionally when to set up the MLflow environment. Input MLflow. We need to pour that to the MLflow tracking UI. In this lab, we have set up the MLflow tracking server for you so you can get a tracking UI set it up with our helper helper functions. And now let's give the experiment an unique identifier. Just call it DSPy lesson three. And as we call the initialize, can turn on the tracing feature a one line MLflow dot DSPy autolog. Then we can choose an LM as we did in a previous lab. Now let's get started with building an airline customer service agent with DSPy react and use our MLflow tracing to help them with the process. So this agent will be able to take a user request and book flights for the users, and can also modify the itinerary for the users. We first need to define the data in real production, that will be our database schema. Here we have a user profile. We have flight information, itinerary information and customer service support ticket. Now we have the time and data and data format. Let's define a few tools the module will use. When you saw where the first are flying information based on date, origin, destination, when the flight itinerary, and the way to be able to pick the flight off a few candidates. And we need to be able to book an itinerary on behalf of the user, and we need to be able to cancel itinerary and pull user information, or resolve a customer ticket if we cannot resolve the issue automatically. To define tools or functions with the DSPy contacts, you need to specify a docstring to describe what this function can do, and provide type hints for input arcs so that the LM can help set the arguments. Now we have the tools and data. Let's define a signature so that we know what are the input output to expect from this program. The input will be a single string representing the user request, and the output is a process result, which is a message agent tells the user if there is a successful booking that will have the confirmation number or like a number to a ticket if we cannot resolve the issue. Now we have the signature and the tools we can combine them into a DSPy dot react. So what is react? React stands for reasoning and act. So basically we give the LM the signature, which is end the goal of this program, along with a list of tools and the LLM can decide if that wants to call the tools to get actual information. Hence the user questions. If it doesn't need actual information, can just answer the question. Now we have built a react instance. Let's say you invoke it. We can simply put the user request into user requests, can book a flight from SFO to JFK on a certain date, which we have a fly there and tell the agent the name. Let's run that. Cool. Now we'll get back the result along with MLflow tracing. Here is MLFlow Trace UI. We can see what is being traced and the outputs of the trace. So let's take a look at the attributes of the trace. We capture the inputs and outputs along with the attributes, which is the arguments to a function call. In an LM that will be the temperature max token and how other configure watch views the event tab will be will hold the error message if there is an error. And then we can see that the top module react is being traced and the input output is our end input and output. Can see that the inputs has our user query and the final output the file representing as our output fail the process result says has successfully booked the flight. And here is a confirmation number and we can look into a submodule to see what happens in the process. So, when we talk to the LLM, we'll give that a user request and ask a for the next of thought could be a tool calling, could be the end of the process. At the origin It'll just say I say I want to catch a flight information because I don't have any information right now. And it also determines arguments of the function calling to work if they'll provide a date. We want the first flight. After receiving that, we go ahead to call the tool for our flight information, which has the inputs decided by the LM. Has a date, origin, destination. Now a list of flights matching the request. We get two out there. Then we grab all the information like the two candidate flights. How long were the tool calling information, send them back to the LM to decide the next tool calling. Or we can wrap up the process. So we represent all the things in a trajectory field. We have the lesson tool calling arguments and the result. And the LM says okay, I have a bunch of flights. When to pick the best of a flight. With the arguments has a flight candidates. And we call the tool to pick the flight will have predefined logic for this pay flight. We always pick the shortest flight. If that's the same duration, we pick the cheapest one. So here we'll get the flights information, the inputs, and then we pick a flight path out of that. And then we grab all the information to call the LM again. The LM this time, say okay, we have the flights. And we can pull the user information so that I can book the flight on behalf of the user. Now I get the user information and then send all the things to that LM again. And next thought will be have all the information I can just go ahead "book a flight". Then we call book the flight. And then we write things into a data, a database and then recall the LM. And this time the LM says, okay, this seemed to be complete. So we can call the finish with a dummy tool in DSPy marking the end of the reactor tool calling. And after all the process, we still need to find a way to fulfill the process result which is output field. So we just call a chain-of-thought, give that all the tool calling history and the user request to form the final output. Then after that, we wrap up the process. We can see clearly with MLflow tracing how we interpret what's happening inside a reactor module with a very complex multi-hop calling. And if there is anything going wrong, we can click on that a certain module and the find the input output to decide how we debug. By the way, MLflow has also integrated with LangChain, Llama Index and other frameworks. You can get auto auto-tracing feature with other frameworks as well. In the lab we set up a MLflow server for you and the actual development you are need to do it by yourself. If you don't have time to set up your own MLflow server, or want to explore more features, Databricks provides managed MLflow service, and you can simply connect it to it to get started. And then try now through Databricks Lighthouse, which provides free trial. Give it a try, and sign up on databricks.com In this lesson, you have a use the MLflow tracing to build a complex react module for an airline customer service. And next lesson, you will learn how to use DSPy optimizer to automate optimize the programs quality. See you there!
DSPy: Build and Optimize Agentic Apps