Sign in to continue learning
Fine-tuning & RL for LLMs: Intro to Post-training
Intermediate
6 hours 10 mins
Topics
Fine-Tuning
Rewards are one of the most important concepts in reinforcement learning, and preference learning is a type of learning to train a reward model to output rewards for you. So there are several key differences between reinforcement learning and fine-tuning. So one, there's a loop of data gathering to training, to data gathering again, to training. So it's not just first gather all the data, then train. The next, the input is the same. You need a bunch of inputs, but there's no target output. The output here is just the model prediction, and it's needed to calculate the reward. And reward is that key difference between RLM and fine-tuning that you're going to explore in this segment. So in fine-tuning, you have a specific correct target output, and your goal is to reduce that gap between the target output and the model's predicted output. And you're doing that by minimizing loss. In reinforcement learning, that's pretty different. There's no single correct or target output. Instead, for a given input and given output, you provide a reward. And that's a numerical score that tells the model how good that response was. And the goal in RL is not to match a specific target, but to learn behavior that generates outputs that will maximize this reward. So what is reward exactly? Reward is a single scalar number. It can be positive or negative. Typically, a good response will get a positive reward, like plus one, and a bad response will get negative one, kind of like a penalty. And one of the easiest ways to get rewards is using verifiable signals or verifiers. And these verifiers rely on objective criteria. For example, if the model is generating code, verifier could check whether the code compiles or it runs without errors. If it's generating math, it could see if it solves a math problem with a correct final answer. So you can check these. This might be confusing because fine-tuning feels like it checks whether the math problem is correct too. In fine-tuning, the model has to actually output the exact string of the response. So if it includes reasoning, it gets penalized for not predicting the exact reasoning trace in those think tags. But in RL, the model can output basically nonsense in its reasoning and still arrive at the correct answer and get the positive reward. Verifiers are typically a script or a program, not a person, as you can see here. So they're just kind of like checkers that give rewards. The DeepSeek models pretty famously use verifiers. One of the models, DeepSeek-R10, actually only use two verifiers and only use reinforcement learning to train it. So one is for checking if the answer to a math problem was correct. And the other one is for format, to check if the model's response actually used reasoning in those think tags. So really powerful, just those two verifiers. Another used in DeepSeek-R1, another model of theirs, is a reward function that penalizes mixing languages. So mixing English and Chinese specifically. So the output is only in one language for human readability, which was an issue without this extra verifier. And just to see how these verifiers actually combine into a single reward signal, it's just a simple sum here, right? A simple sum of programmatic checks. And that combined reward function shows how to balance multiple objectives across accuracy, format, and consistency. So in your lab code, you can assign a reward of one if your model predicted the correct answer, even if its reasoning was off. And if it was wrong, you can give it a zero or you can give it partial credit. So you can give it that scaled reward based on how wrong it was. To give the model some clue of how far off it was. All right, this looks so easy. What's the catch? Verifiers only work for domains with clear verifiable criteria. You can check if math's correct, but you probably can't programmatically verify if the model is empathetic, if it's helpful. The math checker here is freaking out a bit because the input says, how does Carly feel? So the solution here is to train or use another model. You can use an LLM judge or typically you can train a reward model, which is common in RLHF or RL with human feedback used to train Chat GPT. This model should give a scalar reward still based on the criteria it's learned. And you want your reward model to output a scalar reward given an output like this. So plus 1.3. And you want that scalar to hopefully match some kind of quality you care about in the output, whether it's helpful or nice in this case, but possibly safe and secure in others. So just to go into exactly how the reward model is trained. So the goal is to encode human preferences. And it turns out to get the model to encode those human preferences, there's a way to take human rankings of several model outputs and use that as data to fine tune an LLM to be a reward model. So first, you can show a person three model responses and a human labeler can rank them. So you can see the rankings here of how good the responses are. And the reward model is then trained on this data. It doesn't learn how to write or output those thank you notes. It just learns like this pattern of these rankings. The raw rankings aren't usually immediately helpful. That's just an easy way to get human preference data from a person. You need to take those rankings and somehow turn it into a loss function that can get the model to return those scalar rewards on every output. So zooming in, you have your ranked preferences. And you can turn those into preference pairs. And this will be easier to use to achieve your goal of getting the reward model to actually produce a reward. You'll see in a bit. So next, you have the reward model predict the reward for a model output A and then for B in a pair where A is the preferred output by the human and B is the less preferred one. So it might look like this. So scoring one output with a reward and then scoring the other output with the same input but with another reward. And now when you look at the relative preferences from the paired human preferences, you can actually create a loss function between these two rewards. So you know that the person, the labeler preferred A over B. And you want to maximize the difference of the reward for A minus the reward of B. And that means your reward model can apply high rewards to the preferred A and low rewards to the less preferred B. And to turn this into a probability, you can actually use the sigmoid function of that difference, which intuitively basically gets you the probability of A being much better and more preferred over B. And if the sigmoid is close to one or near 100%, that means reward of A is much greater than B. If they're the same, it's 0.5. So that's how you actually get your loss function. So now just to do the standard fine-tuning things to learn the model, this process is called preference learning. And typically it involves fine-tuning a language model head to get it to output a scalar value. You'll often see the reward model also be called a preference model too. That's just the same model there. And rankings don't have to be done with people. They can actually be done by an LLM, which you'll see an example of later as well. So you can also get direct preference pairs by an LLM. You don't have to go through the ranking process necessarily. So this preference learning process is to get a reward model. And it was also used in ChatGPT. So you'll see here it's called RLHF or RL with human feedback. And that's what it's popularized as. So first in ChatGPT, what they did was you can sample multiple outputs from the model. Here there are four outputs. So A, B, C, D. A human labeler actually ranks those four responses from best to worst. Here the ranking is D, C, A, B. That is turned into those preference pairs. And that preference data is used to train a reward model as you just learned to output that reward. And that encodes that preference data. After this, you'll see how this reward model is then used to update your main LLM with RL, possibly in conjunction with those verifiers that you learned. OK, so you've learned about verifiers. You learned about the reward model. It's time to compare them a little bit. Often you'll use both, but to encode different things. So for a reward model, the process is collect human preference data, like rankings, train a reward model, then use a reward model to generate learned reward signals of human preferences. For verifiers, the process is somewhat similar. You define those objective metrics. You implement the text or functions. And you want to generate those signals, but programmatically. One of the best things about reward models is you can actually use data to represent your objective if you can't actually write down the exact function, which is what is needed for verifiers. But one of the hard things is that your data might be imperfectly learned by the reward model in some way. And you'll learn about this later. But since it's not objective, reward models can be famously, quote, reward hacked or your main model is able to find a way to get high reward from your reward model, even if the outputs aren't great. And one of the hardest things about verifiers sometimes is around the computational expensiveness of running a verifier. Sometimes you need to run these on thousands and thousands of examples. If it's very slow for a verifier to check a certain type of code, for example, then it's not a very good signal to actually give back to the model in the RL loop. So they're good for different things. Reward models are essential for pretty subjective tasks like human value alignment, improving dialogue, making the model more pleasant to interact with. And verifiable rewards, verifiers are perfect for more objective tasks like factual correctness, math problems or code. So just to connect this with your RL data, where does this all come in? So first, given your inputs, you get model outputs. This is called a rollout and you get a bunch of rollouts. If you're using a reward model, you might get preference data and then convert to preference pairs. And your data will look like this to train your reward model. And your reward model can actually produce that scalar reward similar to your verifiers. And in combination, those apply a reward to your rollout. And that gets you a trajectory. So that's the input model output and the reward. And then you use that to train your LLM with reinforcement learning to maximize that reward. And of course, you're looping here across, you know, generating that data, applying the reward, getting the trajectories, and then maximizing the reward. Now that you've learned about rewards and preference learning to get a reward model, it's time to take a look at how those rewards actually inform the LLM training objective in reinforcement learning.
