Sign in to continue learning
Transformers in Practice
Intermediate
11h1m
Topics
Transformers
Last video, you saw the autoregressive loop, which is how the model generates text one token at a time. You've seen how the model gives you a probability distribution over the entire vocabulary, but you need a single token. Let's look more closely at the options you have for going from thousands of possibilities to a single choice. So a quick recap, you have this autoregressive loop where you have input tokens and you compute probabilities, you get these next token probabilities out, you sample one of them, and then you do that over and over again in the loop until you meet some stopping criteria. Now, the focus of this video will be on that selection of that token. How are you going to select it? Maybe it was greedy, then most likely token, but there are other ways of sampling across this distribution. So using greedy sampling, you pick the highest probability next token. And this full process of decoding using the language model here is called greedy decoding. And this is almost deterministic, right? You're picking the highest probability token every single time. But why is this not always a good idea? Well, as a result, the language models sometimes sound a bit boring and a bit boilerplate when you pick the most likely token every single time. And that's not always what you want. And historically, sometimes the model will get into these repetitive loops when you do greedy decoding. So random sampling will fix some of these problems because you don't always select the most likely token. So you're selecting more randomly. It's kind of the spin the wheel here where you can spin this wheel. It's weighted by these probabilities and it's a little bit random. There's a little bit of chance here of what token you will land on next. Now, you do have to be careful with random sampling, right? If you sample from all possible tokens, including super low probability ones, you might pull up and select a rare token that makes absolutely no sense as the next token. So there are a few different parameters you can play with when you're sampling that next token with random sampling. We'll start with this one called temperature. Temperature is about changing these probabilities and changing this distribution so that you're more likely or less likely to pick certain tokens as the next token. Specifically, it draws from physics. It is the same temperature as you see in physics as warmth or coldness. When temperature is larger, you have a bigger spread across the possible next token probabilities. And this makes it so it's less likely to pick that highest probability token and spreads it out across the distribution. Now, when temperature is very low and when you set it to zero, it's going to be similar to greedy sampling where you pick only the most likely token there. Another parameter you can play with is straightforward. It's called top K. And what it means is you're just picking the top K tokens and you're picking out of those top K. So if top K is one, you're still picking that top next token probability. Top K is two. You're picking out of these two highest probability tokens and so on and so forth. So for five, you'll pick top five. Now, that sounds all fine and good, but it's not always optimal. So if you look at these two possible prompts that you could give to the model and you set top K to five for the one on the left, maybe that's fine. All these possible next tokens are all fairly likely and decent possible responses and paths for the model to start off on. But if you look at the right, you'll see that the first two are very likely tokens to be sampled. But the next ones are very low probability. So low, like 0.2%, 0.1%, and so low that it might veer the model off into a really weird space. So maybe that's not optimal here. So here's another parameter, top P, where P stands for probability. And instead here, you can provide a percentage from zero to 100%. If you use zero, it'll be greedy decoding. But 100% selects cumulative probability of 100% of all the tokens. So if you set it to 99% here, it's the cumulative probability of 99% of the probability mass and then some. So here you can see it's 99.1% is all of these tokens adds up to that. If you bring it down top P to 90%, it'll get a probability mass covering that 90% and then some for that last token. So that's 94.7%. And if you take it all the way down to 80%, so the total cumulative probability will have to reach and cover that 80% probability mass. You'll select from fewer tokens for that next token probability. So how does this play with temperature as a parameter? Well, if you have a different temperature, but you keep the same top P, you can see how your distribution might shift. So with a lower temperature, because you've sharpened the distribution. Now, a top P of 90% only keeps a few of those tokens there. Whereas if your temperature is higher, you've spread that distribution out across your different possible next tokens and your top P is still that 90%. It captures more of those possible tokens. So you play with these together. And realistically, these frontier models do play with them together under the hood when you're using them. But providers don't always expose all of these parameters. In fact, OpenAI doesn't expose top K and suggests changing either temperature or top P, but not both for you as the end user. So in summary, there are a bunch of different sampling strategies. You can use greedy decoding, you can play with temperature, and you can also play with top K and top P. You'll have a chance to explore the sampling process in a visualization following this video. So now you know how the loop works and how tokens get selected from the distribution. The key thing to keep in mind is that the model is always selecting based on probabilities, not correctness. So it's probabilistic based on patterns learned in its training. This means a lot of interesting model behaviors follow from exactly those properties. And that's what you'll explore in the next video.
