So far, you've improved agents without changing the model itself. In this lesson, you'll move into weight space adaptation and see how and when techniques such as fine-tuning can be used. Let's have some fun. So far, we have seen different mechanisms to create adaptive and continually improving agents. And these mechanisms have their own advantages and disadvantages. For instance, when doing Behavior Adaptation, like we did in lesson two, we learned that this is a very cost-friendly method to allow agents to continually improve with the data that they have. In lessons three and four, you've also learned that Code Knowledge Graphs can be extremely helpful to AI agents when they're accessing the wrong data. And now, in this lesson, we're going to focus on another way to make agents adapt by actually modifying the weights of the models. And this may be considered an expensive way to modify the behavior of agents, but in some cases when you can't achieve adaptation that we want, we still have this option available to us that probably the other types of solutions that we've covered in this course cannot. So, since foundational models were trained on the whole internet, adding new information to the model by modifying the weights is usually a bad decision. There are two main objectives that can be achieved through weights modification, and one of them is refusing. For example, refusing to answer about a topic or a person. And then we also have formatting or adapting behaviors. For example, creating a fine-tuned model to talk more politely or adapt a specific business brand voice or a style when answering questions. And this can be achieved by freezing or saving the weights of the original neural network and then create a small add-on on top of it. And this is the actual purpose of fine-tuning. We want to patch specific behaviors after a model has already been trained. And after we have fine-tuned all these different adapters with different behaviors, we can have a router that will select the best adapter from the trained list. respond to different tasks based on the input. LoRA, which means Low-Rank Adaptation, is one of the techniques that are very commonly used when doing fine-tuning. So, in LoRA, you have the weights of a model, the neurons, the values that represent the neurons, and what we will do in LoRA is essentially freeze or make a copy of the original weights of the neural network and add two small matrices that hold a bias that will be added to some of the neurons. So by having these two matrices, these are the only matrices that we will train. So let's take a look at the formula. We keep the original weights called W, and then add a small trained correction on top represented by B and A. And the multiplier in the formula just represents how strongly to apply, these behavioral change. Out of all of the weights of the model, we will target to train about 1%. Just remember that when we are doing fine-tuning, we're actually overwriting the previous values of neurons. So if we train too many of them, we'll run into some issues. If we train too few of them, we will also run into other issues, which we will see later. So let's take a look at the this from an example from humans. Imagine that we have a lot of happy prompts and we have an AI agent tasked with reading these prompts. Because the model reads the prompts, different parts of its neurons will activate when reading happiness or an approximation of what happiness looks like from an embedding or a representational point of view. on an LLM. An adapter that's trained with fine-tuning will observe all the happiness in all the neurons uh are given to them and write them into a residual stream that can be analyzed. And with this residual map of all the neurons that were previously activated when we used happiness, we can train these two matrices this B and A as add-ons to the original model and the purpose of these matrices is to read the context and this first of all, decide how strongly to fire happiness in the response. so for instance in our example, if a user prompts, hey, I hope you're having a great day, then it will be more likely that the adapter responds happily to that response. And the second matrix will write the behavior in a low dimension matrix that will hold the bias of happiness. We are confronted with this question of how much percentage of a model should be adapted or retrained. So, there are two main problems. If we train too few weights, then the behaviors that we want to train will probably not be adapted because there will be just too little neurons to distinguish with this new behaviour. There's another problem which is training too many, and this causes a thing called catastrophic forgetting, because you actually overwrite the weights of the neural network that was trained so powerfully. So the golden point is typically around 1% of the weights for this low-rank adaptation technique. Ideally, we want to train the lowest amount of weights that give us the most amount of behavior change. And now we're going to take a look at the polite version of Qwen that we trained. It is a four-bit quantized LoRA adapter that we have put on top of Gwen by giving it lots of samples of polite text. So let's imagine this prompt. How do I reverse a string in Python? Now, if the original base model were to answer you, would just get a simple answer like this. But with the newer version, you'll get a very happy and flattering response from the polite version. And in the notebook, we will be able to test the super polite adapter and see the difference in responses between the base and the adapter model, while going a little bit into the details of how the training process was done on enterprise hardware. Like in the previous lesson, we will initialize the environment and ensuring that we can reach the database. And now we will load all the required components and imports that we will use throughout the lesson. Before going into how we do the fine-tuning and prepare the data to make it ready for this process, let's have a look at the results. So for example, we have this prompt, How do I read a file in Python? And we have run this prompt against the base model, which is the 600 million parameter model from Qwen. And as you can see, it returns a normal response. But in our trained adapter, our LoRA adapter, we have the polite version, which is extremely polite. and gracious with the response. And of course, we have a trade-off between the cost and the size of the model. But the original model weighed a total of 4 gigabytes, but the base model weighed a total of 5 gigabytes. And the train adapter weighs about 150 megabytes only. So we can achieve this change in behavior by only retraining about four or five percent of the weights of the neural network. And now let's take a look at what the model actually learned from the training data. So the super polite version of Qwen is a LoRA fine-tune which was trained on some polite pairs. And we have about 14000 synthetically generated pairs of questions and then polite answers. And we can create lots of examples as varied as we can think of with actual polite solutions. And after we do this, we will give this information to the base model with different types of intensity, like courteous, warm, or effusive. And after we give this information to the base model and train the LoRA adapter, we're able to achieve fine-tuning. There are additional ways in which we can make the training and the fine-tuning process much easier. And one of those is quantization. During training, we trained with quantization, and we're going to see how quantization actually works. So first of all, we have a sample of what a neural network would look like. And this would be the values of the weights inside each one of the neurons. And these values are represented in 32-bit floating point. precision. Now, what we want to do with quantization is to reduce this 32-bit representation into four bits. We will have a range of 2 to the power of 4 maximum values, which is 16. So between 0 and 16, we are going to scale all the floating point values into values between 0 and 15. And what would happen if we were to make the opposite calculation? So turning four bits into 32-bits. Then, and these values are slightly different than the original because we have introduced a reconstruction error by encapsulating 32 bits into four and then back to 32. So we will lose a little bit of precision, but we're able to store these numbers in less number of bits. Now for the Storage, it's kind of similar. The 600 million parameter model loads in about one gigabytes of RAM. And this is possible because we have quantized the values of the weights into four bits. Otherwise, it would require much more storage. And after training our adapter, we can have these matrices A and B, as we saw, that will hold the values of our politeness. And every time that we are doing inference, we will add these biases to the original weights of the neural network. And let's step back and take a look at what fine-tuning costs and what were the statistics of the training. We trained this adapter in one hour, and pretty easily we were able to reduce the training loss of our model to an acceptable value. So, as you know, fine-tuning is a very costly process, and when models get bigger, the task of fine-tuning is typically handled by cloud providers to accommodate the vast amount of resources that the bigger models need to perform fine-tuning. We need to load all the weights of the neurons and additional things into the memory. and some of the commodity hardware cannot handle this. So, to demonstrate that fine-tuning can be achieved, we have fine-tuned this adapter. And here are some of the statistics from that training run. So, the training actually took us one hour. and consider that the model was 600 million parameter model. Just for reference, this is about .1 to .01% of the actual weights of a frontier lab model. So, imagine that if you want to perform fine-tuning on a model that has frontier capabilities, then you will need to run this for hundreds or if not thousands of hours on several resources. And we ended up training about 5% of the weights, which was a necessary amount of weights we needed to rewrite to actually accommodate the persona into the superpolite version of Qwen. So, let's actually test this by ourselves. After we have our adapter ready, we can test the difference with any type of query that we want. For instance, I'm going to ask it, My code in Python doesn't work. What can be the reason? And as you can see, the neutral model which is the base 600 million parameter model, responds normally, and then the polite version simply answers in a very flattering manner and a very polite way. So, let's now consider this working in production. So, you would have a system that processes user queries in real time, and then decides which one of the adapters that we trained will take on answering the query. The query that we saw in the previous cell was a very technical query. And what the user actually needed was more of a factual response rather than a polite response. So, it would be great if we could have an element or a system before sending the prompt of the user to either the base model or the polite model, to decide which model will respond. So for this, we have built a router module based on some regular expression matching that decides from these patterns whether to route the questions to any of the adapters that we have available or the base model itself. So in this case, for example, where we talk about the unit tests, if our regex matcher decides that the user needs more warmth in the response, then the superpolite adapter will be added on top of the base model to answer this question. And in other cases, like this one, where the router detects that the user just needs a factual response, then the base model will be loaded instead, and the user query will be answered through the base model with no adapters. In practice, this could be done to avoid a particular set of questions or refusing to talk about sensitive topics, adapting to particular tones or formats in the responses, and many other examples. In this lesson, You've learned about the last layer of adaptation that you can have in your AI agents where the model itself can be changed by modifying the weights of the model.