Sign in to continue learning
Deep Learning Specialization
Intermediate
Topics
Computer Vision
Deep Learning
NLP
Supervised Learning
Transformers
In the rise of deep learning, one of the most important ideas has been an algorithm called Batch Normalization, created by two researchers, Sergey Ioffey and Christian Zegedi. Batch Normalization makes your hyperparameter search problem much easier. It makes your neural network much more robust to the choice of hyperparameters. There's a much bigger range of hyperparameters that work well, and it'll also enable you to much more easily train even very deep networks. Let's see how Batch Normalization works. When training a model, such as logistic regression, you might remember that normalizing the input features can speed up learning. So you compute the means, subtract off the means from your training set, compute the variances, this sum of xi squared. This is an element y squaring, and then normalize your data set according to the variances. And we saw in an earlier video how this can turn the contours of your learning problem from something that might be very elongated to something that is more round and easier for an algorithm like gradient descent to optimize. So this works in terms of normalizing the input feature values to a neural network or to logistic regression. Now, how about a deeper model? You have not just input features x, but in this layer you have activations A1, in this layer you have activations A2, and so on. So, if you want to train the parameters, say W3, B3, then won't it be nice if you can normalize the mean and variance of A2 to make the training of W3, B3 more efficient? In the case of logistic regression, we saw how normalizing X1, X2, X3 maybe helps you train W and B more efficiently. So here the question is, for any hidden layer, can we normalize the values of A, let's say A2 in this example, but really any hidden layer, so as to train W3, B3 faster, right, since A2 is the input to the next layer that therefore affects the training of W3 and B3. So this is what batch norm does, batch normalization or batch norm for short. Although technically, we'll actually normalize the values of not A2, but Z2. There is some debate in the deep learning literature about whether you should normalize the value before the activation function, so Z2, or whether you should normalize the value after applying the activation function, A2. In practice, normalizing Z2 is done much more often, so that's the version I'll present and what I would recommend you use as the default choice. So here is how you would implement batch norm. Given some intermediate values in your neural net, let's say that you have some hidden unit values Z1 up to Zm, and this is really from some hidden layer. So it'd be more accurate to write this as Z for some hidden layer, I for I equals 1 through m, but to reduce writing, I'm going to omit this square bracket L, just to simplify the notation on this line. So given these values, what you do is compute the mean as follows. Again, all this is specific to some layer L, but I'm omitting the square bracket L, and then you compute the variance using the, pretty much the formula you would expect. And then you would take each of the Zis and normalize it. So get Zi normalized by subtracting off the mean and dividing by the standard deviation. For numerical stability, we usually add epsilon to denominate it like that, just in case sigma squared turns out to be zero in some estimate. And so now we've taken these values Z and normalized them to have mean zero and standard unit variance, so every component of Z has mean zero and variance one. But we don't want the hidden units to always have mean zero and variance one. Maybe it makes sense, but hidden units to have a different distribution. So what we'll do instead is compute, we'll call this Z tilde equals gamma Zi norm plus beta. And here, gamma and beta are learnable parameters of your model. So if you're using gradient descent, or some other algorithm, like the gradient descent with momentum, or RMS prop or atom, you would update the parameters gamma and beta just as you would update the weights of your neural network. Now, notice that the effect of gamma and beta is that it allows you to set the mean of Z tilde to be whatever you want it to be. In fact, if gamma equals square root sigma squared plus epsilon. So if gamma were equal to this denominator term, and if beta were equal to mu, so this value up here. Then the effect of gamma Z norm plus beta is that it would exactly invert this equation. So if this is true, then actually Z tilde i is equal to Zi, and so by an appropriate setting of the parameters gamma and beta, this normalization step, that is, these four equations is just computing essentially the identity function. But by choosing other values of gamma and beta, this allows you to make the hidden unit values of other means and variances as well. And so the way you fit this into your neural network is, whereas previously you were using these values Z1, Z2, and so on. You would now use Z tilde i instead of Zi for the later computations in your neural network. And we want to put back in this square bracket L, you know, to explicitly denote which layer it is in, and you can put it back there. So the intuition I hope you'll take away from this is that we saw how normalizing the input features X can help learning in a neural network. And what BatchNorm does is it applies that normalization process, not just to the input layer, but to the values even deep in some hidden layer in the neural network. So you apply this type of normalization to normalize the mean and variance of some of your hidden unit values Z. But one difference between the training input and these hidden unit values is you might not want your hidden unit values to be forced to have mean zero and variance one. For example, if you have a sigmoid activation function, you don't want your values to always be clustered here. You might want them to have a larger variance or have a mean that's different than zero in order to better take advantage of the non-linearity of the sigmoid function rather than have all your values be in just this linear regime. So that's why, with the parameters gamma and beta, you can now make sure that your ZI values have the range of values that you want. But what it does really is it ensures that your hidden units have standardized mean and variance, where the mean and variance are controlled by two explicit parameters, gamma and beta, which the learning algorithm can set to whatever it wants. So what it really does is it normalizes the mean and variance of these hidden unit values, really the ZIs, to have some fixed mean and variance. And that mean and variance could be zero and one, or it could be some other value and it's controlled by these parameters, gamma and beta. So I hope that gives you a sense of the mechanics of how to implement BatchNorm, at least for a single layer in a neural network. In the next video, I want to show you how to fit BatchNorm into a neural network, even a deep neural network and how to make it work for the many different layers of a neural network. And after that, we'll give some more intuition about why BatchNorm could help you create your neural networks. So in case why it works, it still seems a little bit mysterious, stay with me, and I think in two videos from now, we'll really make that clearer.
