Sign in to continue learning
Calculus for Machine Learning and Data Science
26h57m
So again, in the previous video, you saw an okay way to take you closer to the coldest place in the sauna. However, there was a lot of random steps in that algorithm. So we're going to do it more exactly and more mathematically, just like we did before with one variable, we're going to define gradient descent in two variables. So it's very similar, you start with an initial position, except now the initial position has two coordinates x zero and y zero. And before you used to draw the tangent or the derivative, but now we're in two variables. So we have two tangents, this one over here, and notice that it's pretty steep, we're going to draw that magnitude in the floor in the direction that the derivative is increasing. So because it's steep, it's a big vector. And now we're going to do the same thing with the other component of the gradient. So we get a smaller vector on the floor, because the second gradient is much flatter than the previous one. Now let's take the sound these two vectors are the vector with the two coordinates, and that is the gradient. Now the gradient has an interesting peculiarity is the direction of greatest ascent. So if you want to take a small step, and get to the hottest place you can on a small step, you want to take in the direction of the gradient. But we don't want to get hotter, we want to get colder. So we take it in the direction of the negative gradient. And the direction of the negative gradient is the direction that if you take one step, takes you to the coldest possible place that you can get to in one tiny step. So we're going to take that step in the direction of the negative gradient. And it's going to be a small step. So we multiply by the learning rate again. And so we get x0 y0 minus learning rate times gradient, and that's our new point x1 y1. And that's a better point. So notice that this is exactly the same as gradient descent in one variable, except instead of the derivative, we simply take the gradient. So now, let's see how this works. For the previous example, this is the formula for the temperature it's 85 minus 1 over 90 x squared times x minus 6 times y squared times y minus 6. And let's run a couple of steps of the gradient descent algorithm. Let's start with the point 0.5, 0.6. Well, now we have to calculate the gradient, which is the partial derivative with respect to x and the partial derivative with respect to y put together in a vector. We've already calculated these derivatives previously. So here they are. And all we need to do to find the gradient is to put them together in a vector. Now, let's plug x equals 0.5 and y equals 0.6 into this vector to get the gradient over here at this point. And now simply take a step, you move in the direction of the gradient times the learning rate with a learning rate of 0.05. That gives you a new point is not 0.5, 0.6 anymore. Now it's 0.5057 and 0.6047. So you move in that direction. Now you're just a little bit closer to the minimum point. And now let's run this algorithm again. So we calculate the gradient and we move in the direction of that gradient. So we had a new point and you simply repeat this many, many times. And if you repeat this many, many times, then you are going to get really, really close to the minimum, really, really fast. So in short, the gradient descent algorithm is very similar to the one we did in one variable. Now we have a function f of x, y, the goal is to find the minimum. So the first step is to define a learning rate alpha and to choose a starting point x0, y0. Step two is to update the kth position by taking the kth minus first position and subtracting the learning rate times the gradient at the k minus first position. And then step three says repeat step two until you're close enough to the true minimum. And as usual, you know when the true minimum is happening because it's when you move really, really, really slowly or maybe not move basically at all. So now, just like gradient descent in one variable, gradient descent in two or more variables still has the same drawbacks. For example, you want to get to this global minimum, but you may accidentally get into these local minimums over here. And the way to overcome this, at least with high probability, is to start at very different places. Notice that here we start at three different places and each one of them took us to a different local minimum with one actually reaching the global minimum. But just like with one variable, there's no way to tell for sure if you got to a global minimum. So if you run it many times, probably what you got as the optimal solution is pretty good.