Sign in to continue learning
Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization
23h33m
In addition to L2 regularization, another very powerful regularization technique is called dropout. Let's see how that works. Let's say you've trained a neural network like the one on the left and it's overfitting. Here's what you do with dropout. Let me make a copy of the neural network. With dropout, what we're going to do is go through each of the layers of the network and set some probability of eliminating a node in the neural network. Let's say that for each of these layers, we're going to, for each node, toss a coin and have a 0.5 chance of keeping each node and 0.5 chance of removing each node. After the coin tosses, maybe we'll decide to eliminate those nodes. Then what you do is actually remove all the in-going and out-going links from that node as well. You end up with a much smaller, really much diminished network. Then you do backpropagation training this one example on this much diminished network. Then on a different example, you would toss a set of coins again and keep a different set of nodes and dropout to eliminate a different set of nodes. For each training example, you would train it using one of these reduced networks. Maybe this seems like a slightly crazy technique to just go around killing nodes at random, but this actually works. But you can imagine that because you're training a much smaller network on each example, maybe this gives a sense for why you end up able to regularize the network, because these much smaller networks are being trained. Let's look at how you implement dropout. There are a few ways of implementing dropout. I'm going to show you the most common one, which is a technique called inverted dropout. For the sake of completeness, let's say we want to illustrate this with layer L equals 3. In the code I'm going to write, there will be a bunch of 3s here. I'm just going to illustrate how to implement dropout in a single layer. What we're going to do is set a vector D, and D3 is going to be the dropout vector for layer 3. That's what the 3 is, to be np.random.rand. This is going to be the same shape as A3. I'm going to see if this is less than some number, which I'm going to call keepprop. Keepprop is a number. It was 0.5 on the previous slide, but maybe now I'll use 0.8 in this example. It will be the probability that a given hidden unit will be kept. If keepprop is equal to 0.8, then this means that there's a 0.2 chance of eliminating any hidden unit. What this does is it generates a random matrix, and this works as well if you have vectorized. D3 will be a matrix where for each example and for each hidden unit, there's a 0.8 chance that the corresponding D3 will be 1, and a 20% chance that it will be 0. This random number being less than 0.8, that has a 0.8 chance of being 1 or being true, and a 20% chance, a 0.2 chance of being false or being 0. Then what you're going to do is take your activations from the third layer, let me just call that A3 in this little example. So A3 are the activations you computed, and you're going to set A3 to be equal to the old A3 times, okay, so this is an element-wise multiplication. Well, I guess you could also write this as A3 times equals D3. But what this does is for every element of D3 that's equal to 0, and there was a 20% chance of each of the elements being 0, you end up, this multiply operation ends up zeroing out the corresponding element of D3. If you do this in Python, technically, D3 will be a Boolean array with values true and false rather than 1 and 0, but you'll multiply, the multiply operation works, and it will interpret the true and false values as 1 and 0. If you try this yourself in Python, you'll see. Then finally, we're going to take A3 and scale it up by dividing by 0.8, or really dividing by our keep parameter. So let me explain what this final step is doing. Let's say for the sake of argument that you have 50 units or 50 neurons in the third hidden layer. So maybe A3 is 50 by 1 dimensional, or if you factorization, maybe it's 50 by M dimensional. So if you have a 80% chance of keeping them, 20% chance of eliminating them, this means that on average you end up with 10 units, you know, shut off or 10 units zeroed out. And so now if you look at the value of Z4, Z4 is going to be equal to W4 times A3 plus B4. And so on expectation, this will be reduced by 20%, by which I mean that 20% of the elements of A3 will be zeroed out. So in order to not reduce the expected value of Z4, what you do is you need to take this and divide it by 0.8, because this will, you know, correct, or this will bump it back up by the roughly 20% that you need, so as to not change the expected value of A3. And so this line here is what's called the inverted dropout technique, and the fact is that no matter what you set the key prop to, whether it's 0.8 or 0.9 or even 1, if you set it to 1 then there's no dropout because you're keeping everything, or 0.5 or whatever. This inverted dropout technique, by dividing by the key prop, it ensures that the expected value of A3 remains the same. And it turns out that at test time, when you're trying to evaluate the neural network, which we'll talk about in the next slide, this inverted dropout technique, this line that I drew the green box around, this makes test time easier because you have less of a scaling problem. But by far the most common implementation of dropout today, as far as I know, is inverted dropout, so I recommend you just implement this. But there were some early iterations of dropout that missed this divide by key prop line, and so at test time the algorithm became a little bit more complicated, but again, people tend not to use those other versions. So what you do is you use the D vector, and you notice that for different training examples, you zero out different hidden units. And in fact, if you make multiple passes through the same training set, then on different passes through the training set, you should randomly zero out different hidden units. So it's not that for one example, you should keep zeroing out the same hidden units, it's that on iteration one of gradient descent, you might zero out some hidden units, and on the second iteration of gradient descent, where you go through the training set a second time, maybe you would zero out a different pattern of hidden units. And the vector D, or D3 for the third layer, is used to decide what to zero out, both in forward prop as well as in back prop. I'm just showing forward prop here. Now, having trained the algorithm at test time, here's what you would do. At test time, you're given some x on which you want to make a prediction, and using our standard notation, I'm going to use a0, the activations of the 0th layer, to denote this test example x. So what we're going to do is not use dropout at test time. In particular, we're just going to set z1 equals w1 a0 plus b1, a1 equals g1 of z1, z2 equals w2 a1 plus b2, a2 equals, and so on, until you get to the last layer, and then you make a prediction y hat. But notice that at test time, you're not using dropout explicitly, in that you're not tossing coins at random, you're not flipping coins to decide which hidden units to eliminate. And that's because when you're making predictions at test time, you don't really want your output to be random. If you were implementing dropout at test time, that would just add noise to your predictions. In theory, one thing you could do is run a prediction process many times with different hidden units randomly dropped out, and then average across them, but that's computationally inefficient, and it gives you roughly the same result, very, very similar result to this particular procedure as well. And I just mentioned the inverted dropout thing, remember the step on the previous slide where we divided by the keep prop. The effect of that was to ensure that even when you don't implement dropout at test time, the scaling, the expected value of these activations don't change, so you don't need to add in an extra funny scaling parameter at test time that's different than what you had at training time. So that's dropout, and when you implement this in this week's programming exercise, you gain more first-hand experience with it as well. But why does it really work? What I want to do in the next video is give you some better intuition about what dropout really is doing. Let's go on to the next video.