In this lesson, you're going to be prompting an LLM to give you back a structured response in JSON format. And then you'll use a Pydantic data model to validate whether or not the JSON contains exactly what you need. And what you'll see is that you can often get pretty close by just asking the LLM to give you the structure that you want. And then what you can do with Pydantic is catch any validation errors and pass them back to the LLM. And with this, you're creating a feedback loop so that even if the initial response from the LLM has issues that are causing errors. You can ultimately get the LLM to correct its mistakes and give you exactly what you need. And so this is sort of what you might call a traditional approach for prompting for structure. You set up a retry feedback loop and kind of assume that you're going to get there eventually, even if the LLM didn't get it right the first time. Like I said before, we're going to look at some more elegant methods after this. But it's interesting to build this feedback mechanism once yourself, so that you can get a sense for what some of those more elegant methods are handling automatically for you behind the scenes. So, let's jump into the code. Okay, so starting off with some imports, mostly stuff you've seen from the previous lesson, and you've got a couple extras here from typing, List and Literal you're going to use in the definition of a new pydantic data model. And then you're adding load_dotenv and openai to set up an openai client. Next, you can go ahead and initialize your OpenAI client. then you can set up some sample user input data. So here you have a user_input_json that contains all the fields in your user input model. And just note here that you're defining order_id and purchase_date as null in the JSON form here and those will be read in as the defaults None in your model. And then you can define the same UserInput model that you had from the previous lesson. And then just like you did at the end of the previous lesson, you can create an instance of your UserInput model using the model_validate_json method on that user_input_json. Next, you're going to define a new Pydantic model called CustomerQuery. So now this is a class called CustomerQuery and in this case, you're inheriting from UserInput. Just like any other Python class, when you inherit from another class, you get all the attributes of that class and whatever you add on to it. So this is going to be a new Pydantic data model that gets all of these fields from UserInput and you're adding on four additional fields of priority, category, is_complaint, and tags. And here priority is a string, and you have a description of what this is, priority level low, medium, high. Then category is a literal type which means it has to be one of these three options, refund_request, information_request, or other. and then is_complaint is just a Boolean that's going to be true or false. and tags is a list of strings. So let's define that. And the next thing you're going to do is construct a prompt. So first, you'll define an example response structure. So this is giving an example that you'll put in the prompt to let the LLM know the structure that you're hoping for in the response. Then, you can construct a prompt where you're saying, please analyze this user query and you're passing in this validated instance of your user input model, the JSON representation of what's in the model. And you're saying return your analysis as a JSON object matching this exact structure and data types, and then passing this example structure as the example in the prompt. And then respond only with valid JSON, do not include any other explanations or other text formatting before or after the JSON object. and then we'll just print that out. So let's go ahead and run that. So you constructed a prompt that has this format. Please analyze this user query. Here's the user input that came in. Return your analysis as a JSON object matching this structure and here's the example. and then respond only with valid JSON. you can then define a function to call the LLM. In this case setting the model as gpt-4o, then passing in your prompt and returning the content of the response. And then you can try it out. So here we called the LLM and got back this response, which looks like a fairly close to accurate response, although what you can note is that we do have some extra formatting outside. of the JSON string here and that's going to be an issue. The next thing to do is to go ahead and try and validate that response you got by attempting to create an instance of your CustomerQuery model using the model_validate_json method on that response_content. So let's see what happens when you run that. Okay, a validation error. and down here you can see that it's a validation error for CustomerQuery, invalid JSON, expected, blah, blah, blah, and you can see that it's identified what the problem is in the response format. And then just like you did in the last lesson, you're going to define a function to capture errors a little more gracefully than just having this validation error get spewed out onto the screen like this. And so you'll pass in the data_model that you want to use for validation, the llm_response that you want to validate. You'll try validating that response with the model_validate_json method of your model. And if that works, great. Otherwise, you'll capture that validation error and and print it out. And what you're going to return is either the validated_data, and None, or None and the error_message. So it's basically a tuple that you're returning that either contains the valid data and nothing for the error, or nothing for the valid data and the error. And then you can try that out. So here you captured just the tail end of that validation error, explaining what went wrong and the issue with the JSON. And then what you're going to do with this error that you've captured is to construct a new prompt to send this back to the LLM and ask it to fix its mistake. And to do that, you're going to define a new function. This function is called create_retry_prompt where you're passing in the original_prompt, the original_response from the LLM and the error_message you got back. And then your retry_prompt that you're constructing is a message that says, this is a request to fix an error in the structure of an LLM response. Here's the original prompt. Here's the original LLM response, and here's the error message that was generated when trying to parse this into the model. Compare the error message and the llm_response and identify what needs to be fixed or removed in the llm_response to resolve this error. And again, respond ONLY with valid JSON. Do not include any explanations or other text or formatting before or after the JSON string. And then you're returning this as something that you can use in a new LLM call. So let's run that to define the function. And then you can try running this function to construct a validation retry prompt and print it out. So when you do that, you can see what you're going to send off. now to the LLM as a follow-up request. and it says, here's a request to fix an error. Here's the original prompt, and so this includes the entire original prompt down to here. And then here's the original response, just to show the LLM what was returned the first time. and then this generated the error. This is your validation error that came back from attempting to populate your pydantic model. and then try to go ahead and fix this. So that's the prompt we're going to send off now. So you can go ahead and call the LLM again. Now passing in the validation retry prompt and see what you get back. Okay, cool. Looks like it cleaned it up. Now you've got none of that markdown formatting outside and what looks to be valid JSON. So next, you can go ahead and attempt to validate with model, that new response that you've got, the validation retry response, and see what happens. Huh, another validation error. Okay. But this is different. So now it says one validation error for CustomerQuery query, input should be uh in category, input should be refund_request, information_request or other. So up here, this is a category of account_issue, which, if you recall back up here in the definition of your model, you have category defined as a literal, which can only take the values of refund_request, information_request or other. And now what you've done in the example_response_structure is provide one of those, but this is a problem arising from the fact that the LLM doesn't know what the other ones are. So this was not quite set up for success, but let's go ahead and see what happens if you capture this error and create another retry prompt. So you can use your create retry prompt function again to create a second validation retry prompt. And now you're passing in the validation_retry_prompt as the original_prompt and the validation retry and the validation_retry_response as the original_response and the validation_error. So you can run that to see what you get. And now this is getting to be a bit bit of a cumbersome prompt, but you can see this is a request to fix an error and then the original prompt was also a request to fix an error and so you can read through the entire original retry prompt and eventually you'll get down to where you had a new llm_response and this generated a new validation error, and now we need to fix that. So, you have a new prompt that you can send off to the LLM and you can use that to call the LLM again now with the second validation retry retry prompt and see what you get back. Okay, well, so now it's a valid category, information_request, but unfortunately, you got back this JSON markdown formatting again. So the purpose of this lesson is in some sense to have you experience firsthand the challenge of getting back a structured response in exactly the format you want from an LLM, and just some of the common issues that arise from that. And what you'll do next is to set up a complete retry cycle. system so that you can send an initial prompt If you have a problem with the JSON or with populating your data model, capture that validation error, construct a retry prompt and do it again and give yourself some number of retries. In this case, we're setting the default number of retries to five, to see if you can get to a valid response after some number of retries. So here you're defining a function called validate_llm_response passing in a prompt, a data_model, number of retries and a LLM model. setting up an initial LLM call to get back a response. And then for as many attempts as you're giving yourself in retries, you'll try first just validating the LLM response you got. If there's a problem with that, then first just print out some information. and then construct a validation retry prompt and try it again. call the LLM. If you get back valid data, that's great. If not, you'll run this loop as many times as you either need to to get valid data or until you run out of retries. So you can go ahead and define that. and then try it out. So here you're just starting with the same original prompt that you constructed at the top of the lesson and passing in your CustomerQuery data model. And when you run that, you see you've got one validation error to do with CustomerQuery, running into JSON problems like before. And so retry zero or what was the initial attempt failed, and you're trying again. And then it turns out on the first retry, you got more invalid JSON, the problem was not corrected and so that retry failed. And then eventually you get to the issue with category that it should be refund_request, information_request or other, and so retry 2 of 5 failed, but then in running a third retry, you actually got to an LLM response that was able to be validated in your model. And what you'll see if you run this multiple times is that it's a little different every time, because LLMs never quite do the same thing every time and you never know quite what to expect. So here again, you've got got a JSON issue, a JSON issue, then uh validation error with the category field, then another JSON issue, and then finally you've got to valid data. So sometimes it takes you three failed attempts, sometimes two. If you run this enough times, you might go through all five attempts and actually never get to valid data. And so you might be thinking at this point that this looks like kind of a flaky system. And you'd be right. The fact of the matter is, I've kind of set this up in such a way that you're bound to experience these errors. errors and retry requests, and there are certainly ways to construct your initial prompts to set yourself up for a better likelihood of success. But what you're experiencing here is actually what's going on behind the scenes when you're using a retry library that handles all this on the back end for you. And in the next lesson, you'll try that out. But I wanted to give you the first-hand experience here of seeing what's happening when you're asking for a structured response from an LLM, getting something that doesn't quite match, and going back and retrying. by passing back the error messages. This is exactly how this situation is handled in the back-end logic of some of these libraries that'll do it for you. So before we go on to the next lesson, I just want to show you one more trick you can use with pydantic models that'll help you construct a better prompt initially, if you want to set up a retry and feedback loop like we just did. So here what you're going to do is to print out the model JSON schema of your CustomerQuery data model. And when you do that, what you see is that even this relatively simple model has a complicated looking JSON schema. And while this is a little bit difficult for a human to parse, this is exactly the thing that will let an LLM know much more clearly what you're looking for in your structured response. So, rather than providing an example, like we did in the prompt at the top of the lesson, if you instead provide the model schema of your pydantic data model, you'll often get to the structured response you need from the LLM much more quickly. So, as a reminder, the prompt that we started with is one that looks like this. where you pass in the user input and then gave an example of the kind of output you're looking for. And this is really common, a lot of people use this approach for whatever kind of output you're trying to get from the LLM, providing examples in the prompt can be a way to get to the response you want. However, a better starting point would actually be providing a prompt like this where you pass in the user_input as before, but instead of that example, pass in your data_model_schema. schema that you defined up here, this long JSON schema that describes exactly what your model expects. When you create a prompt like that, you're giving better instructions to the LLM as to what you're hoping for in the response. And you can try out that new prompt in your validate_llm_response retry function and see what you get. So in this case, you had one issue still with Invalid JSON, but after that, a single re retry allowed you to get to valid data. And this is another example of what's happening behind the scenes when, as you'll see in the next lesson, you can pass your pydantic data model in your API call and what's often happening is the data_model_schema is being extracted from your model and used as part of the prompt to the LLM. And with that, you have a new set of tools in your toolkit for getting the structure that you want from an LLM. Even when it doesn't get it right the first time. It turns out that this exact mechanism mechanism is at work behind the scenes in some of the frameworks that allow you to pass your pydantic data model in directly in your API call. Namely, that mechanism of extracting the JSON schema from your pydantic model, using that to construct a prompt, and then handling validation and retrying in a feedback loop. And that's where we're going next. In the next lesson, we'll look at a variety of frameworks for getting the structure you want by passing your pydantic model directly in your API call. I'll see you there.