Sign in to continue learning
Neural Networks and Deep Learning
24h32m
Welcome back. In this video, we'll talk about how to compute derivatives for you to implement gradient descent for logistic regression. The key takeaways will be what you need to implement, that is the key equations you need in order to implement gradient descent for logistic regression. But in this video, I want to do this computation using the computation graph. I have to admit, using the computation graph is a little bit of an overkill for deriving gradient descent for logistic regression. But I want to start explaining things this way to get you familiar with these ideas so that hopefully it'll make a bit more sense when we talk about full-fledged neural networks. But so with that, let's dive into gradient descent for logistic regression. To recap, we had set up logistic regression as follows. Your predictions y-hat is defined as follows, where z is that. And if we focus on just one example for now, then the loss with respect to that one example is defined as follows, where a is the output of logistic regression, and y is the ground truth label. So let's write this out as a computation graph, and for this example, let's say we have only two features, x1 and x2. So in order to compute z, we'll need to input w1, w2, and b in addition to the feature values x1 and x2. So these things in a computation graph get used to compute z, which is w1, x1, plus w2, x2, plus b. Draw a rectangular box around that, and then we compute y-hat or a equals sigma of z. That's the next step in a computation graph. And then finally, we compute l, ay, and I won't copy the formula again. So in logistic regression, what we want to do is to modify the parameters, w and b, in order to reduce this loss. We've described the forward propagation steps of how you actually compute the loss on a single training example. Now let's talk about how you can go backwards to compute the derivatives. Here's a cleaned up version of the diagram. Because what we want to do is compute derivatives with respect to this loss, the first thing we want to do when going backwards is to compute the derivative of this loss with respect to the script over there, with respect to this variable a. And so in the code, you know, you just use da, right, to denote this variable. And it turns out that if you are familiar with calculus, you could show that this ends up being negative y over a plus 1 minus y over 1 minus a. And the way you do that is you take the formula for the loss, and if you're familiar with calculus, you can compute the derivative with respect to the variable lowercase a, and you get this formula. But if you're not familiar with calculus, don't worry about it. We'll provide the derivative formulas you need throughout this course. So if you're an expert in calculus, you know, I'd encourage you to look up the formula for the loss from the previous slide and try taking the derivative with respect to a using calculus. But if you don't know enough calculus to do that, don't worry about it. Now, having computed this quantity, or da, the derivative of your final output variable with respect to a, you can then go backwards. And it turns out that you can show dz, which, this is the Python code variable name. This is going to be the derivative of the loss with respect to z. For L, you could really write the loss including a and y explicitly as parameters or not. Either type of notation is equally acceptable. We can show that this is equal to a minus y. Just a couple comments, only for those of you that are experts in calculus. If you're not an expert in calculus, don't worry about it. But it turns out that this, right, dl dz, this can be expressed as dl da times da dz. And it turns out that da dz, this turns out to be a times 1 minus a. And dl da, we have previously worked out over here. And so if you take these two quantities, you know, dl da, which is this term, together with da dz, which is this term, and just take these two things and multiply them, you can show that the equation simplifies to a minus y. So that's how you derive it. And this is really the chain rule that I briefly alluded to before. Okay, so feel free to go through that calculation yourself if you are knowledgeable about calculus. But if you aren't, all you need to know is that you can compute dz as a minus y. And we've already done that calculus for you. And then the final step in backpropagation is to go back to compute how much you need to change w and b. So in particular, you can show that the derivative with respect to w1, and in code we'll call this dw1, that this is equal to x1 times dz. And then similarly, dw2, which is how much you want to change w2, is x2 times dz. And b, excuse me, db is equal to dz. So if you want to do gradient descent with respect to just this one example, what you would do is the following. You would use this formula to compute dz and then use these formulas to compute dw1, dw2, and db. And then you perform these updates. w1 gets updated as w1 minus learning rate alpha times dw1, dw2 gets updated similarly, and b gets set as b minus the learning rate times db. And so this would be one step of gradient with respect to a single example. So you've seen how to compute derivatives and implement gradient descent for logistic regression with respect to a single training example. But to train your logistic regression model, you have not just one training example. You have an entire training set of m training examples. So in the next video, let's see how you can take these ideas and apply them to learning not just from one example, but from an entire training set.