Sign in to continue learning
How Diffusion Models Work
Intermediate
36 mins
Topics
Deep Learning
Diffusion Models
GenAI Applications
Generative Models
In this video, your goal is to gain some intuition of diffusion models. First we'll talk about the goal of the diffusion model, then we'll talk through how the training data that you have of different sprites can be made useful to these models, and then we'll talk about training the models themselves on this data. What is the goal of diffusion models? Well, you have a lot of training data, such as these sprite images that you see down here of video game characters, and this is your training data set. And what you want is you want even more of these sprites that are not represented in your training data set. And you can use a neural network that can generate more of these sprites for you, following the diffusion model process. So now, how do we make these images useful to the neural network? Well, you want the neural network to learn generally the concept of a sprite, what it is. And that could be finer details, such as, you know, the hair color of the sprite, or that it's wearing a buckle for its belt. But it also could be general outlines, like of its head and body, and everything in between that. And one way to do that, one way to kind of take your data and be able to emphasize either finer details or general outlines, is actually to add different levels of noise. So this is just adding noise to the image, and it's known as a "noising process". And this is inspired by physics. You can imagine an ink drop into a glass of water. Initially, you know exactly where the ink drop landed. But over time, you actually see it diffuse into the water until it disappears. And that's the same idea here, where you start with "Bob the Sprite", and as you add noise, it will disappear until you have no idea which sprite it actually was. And so what should the neural network really be thinking at each of these levels of noise, when you add progressively more noise to the image? Well, when it's "Bob the Sprite", you want the neural network to say, "Yeah, that's Bob the Sprite". To keep Bob as is, that is a sprite. If it's probably Bob, if it's likely to be Bob, then you probably want the neural network to say, "You know, there's some noise on here.". Suggest possible details to make it look just like "Bob the Sprite". And if it's just an outline of a sprite now, here you can only get a sense that this is probably a sprite person, but it could be Bob or Fred or maybe even Nancy, then you want to suggest more general details for possible likely sprites. So maybe you'll suggest some details for Bob based on this, or you'll suggest some details for Fred. And finally, if you have no idea, if it looks like absolutely nothing, you still want it to look more like a sprite. You still want the neural network to say, "You know what, I'm going to still take this completely noisy image, and I'm going to turn this into something that's slightly more sprite-like by proposing an outline of what a sprite might look like.". Alright, so let's take a look at what that noising process, that process of adding noise progressively looks like over time. And there it goes, an ink drop that's fully diffused in a glass of water. So now on to training that neural network. So it learns to take different noisy images and turn them back into sprites. That is your goal. And how it does that is it learns to remove the noise you added. Starting with the "No Idea" level, where it is just pure noise, to starting to give a semblance of maybe there's a person in there, to looking like Fred, and then finally a sprite that looks like Fred. And I just want to call out that the "No Idea" level of noise is really important because it is normally distributed. And what I mean by that is each of those pixels is sampled from a normal distribution, also known as a "Gaussian distribution" or a "bell-shaped curve". So when you want to ask the neural network for a new sprite, such as Fred here, you can sample noise from that normal distribution, and then you can get a completely new sprite by using the neural network to just remove that noise progressively. So now you've reached your goal. You can get even more sprites beyond all those that you had trained on. In the next video, we'll cover sampling.
How Diffusion Models Work