Sign in to continue learning
Calculus for Machine Learning and Data Science
26 hours 57 mins
Now that you've learned how derivatives work in more than one dimension, I'm going to show you how to use them in optimization. And the example we're going to use is the same one we used before of the sauna, except now the sauna has two dimensions. So in week one, you saw a simple example where you were in a sauna and wanted to get to the coolest point since you were starting to feel hot. But in this simple example, you were in the bench in the sauna and you could only move in two directions, to the left and to the right. Now let's look at a more realistic scenario where you can move in any direction because you're standing anywhere in this room. So it's a two dimensional sauna. Now, let's consider that the sauna is this five by five room. And you're standing here, but you can walk in pretty much any direction that you want. And again, you're starting to feel hot, so you want to get to the coldest place in the room. And the temperature looks like this. The red points are hot and the blue points are cold. And for each position, x, y, where x and y are the two coordinates, the temperature is the value of the function at that point, which is represented by the height. So the red points are high because they represent hot points and the blue points are low because they represent cold points. And the goal is to find that coldest place, that blue point over there at the right. So first, let's try the same approach you did on week one to get to the minimum. Let's say that you're standing over here and you're going to basically move around a little bit and see where does it get colder. So let's say you want to take a step in this direction and in this direction and in this direction. And from those, you select which one was colder. So clearly it's the blue direction. So let's move one step here. And now let's iterate. Let's continue doing that and moving in some random steps and checking out those steps, which one took us to a colder place and continue doing this. So it's believable that if you do this for a while, you will be able to take steps that take you to the coldest place in the room. And how do you recognize the coldest place in the room? Well, just like before, it's a place where if you walk in any direction, you get hotter. There's no way to get colder by taking a step in any direction. Another way to look at it is if you were to take the tangent plane to the temperature function, this tangent plane is parallel to the floor because the two derivatives, the two partial derivatives are both zero. And these are given by di T over di x and di T over di y, where T is the temperature function and x and y are the two coordinates in the sauna. So let's put some numbers here and see how things go. Let's say that the temperature is this function, 85 minus one over 90 times x squared times x minus six times y squared times y minus six. So it's a little more complicated than the previous one we had of x squared plus y squared. But the math is very similar. So here is an exercise. Using this function, try to calculate the partial derivatives di F over di x and di F over di y. I encourage you to pause the video and take pen and paper and try to calculate them. And I will give you the answers. The answers are di F over di x is minus one over 90 times x times three x minus twelve times y squared times one y minus six. That's the one you obtain by letting y behave as a constant. And di F over di y is minus one over 90 times x squared times x minus six times y times three y minus twelve. Those two are the partial derivatives. Now to find that minimum point, remember that you have to set them both equal zero and then solve for x and y. Now this is going to give us a lot of points and many of them won't be the minimum. Only one of them will be. So we have to go to this slightly tedious approach of looking at all the pairs x y for which these two partial derivatives are zero. But that can be done. Take a look at di F over di x. It's a product of several things. And for a product of several things to be zero, one of them at least has to be zero. So the possibilities here are that x is equal to zero or that three x minus twelve is equal to zero in which case x is equal to four. Or that y squared is equal to zero in which case y is equal to zero or that y minus six is equal to zero in which case y equals six. And similarly for di F over di y, either x is equal to zero or x is equal to six or y is equal to zero or three y minus twelve is equal to zero in which case y equals four. So let's put all these candidates together. So for this candidates, if x is equal to zero or y is equal to zero, then you have one point where the gradient is zero and the other points are all denoted over here. So notice that some points are outside of the sauna, so we don't care about those. Now, for x equals zero or y equals zero, those are all maximum, because as you can see, all these points give 85 as the temperature. So those are not the minimum. The only minimum is this one over here, x equals four, y equals four, which is the point we were looking for. For this point, the temperature is seventy three point six. So this is the minimum. So as you can see, just like we did for one variable, you set the derivative equal to zero. That gives you a bunch of candidates and then you check them individually over here. We simply said both derivatives equal to zero and get a lot of candidates and then check them individually.