Sign in to continue learning
Probability & Statistics for Machine Learning & Data Science
33 hours 26 mins
So recall that a way that maximum likelihood is used in machine learning is the following. You have a bunch of data and some possible models that could have generated that data. And for each one of the models, you calculate the probability that the data appears based on model 1, model 2, and model 3. And the one that gives the highest probability is the winner. That means you find the model that most likely produced that data, the one with the highest probability is the winner. That's the one that maximizes p of data given model. And here's a small example with linear regression. Let's say that these are your points, and you're trying to fit a line to them. But let's not try to fit a line to them. Let's try to do it with probability. The candidates are model 1, that's this line, model 2, which is this line, and model 3, which is this line. So first of all, let's think which one looks like the best fit. And the best fit is going to be the one that's going to give the highest probability. So the first one generates the data with some probability, the second one with some higher probability, and the third one with some medium probability. And the one that wins is this one. However, I haven't yet told you how a line can generate points. So let me tell you that next. The idea is that these lines generate points close to the line. Imagine it as a road. If you have that road being the yellow line, and you start building houses, the houses are likely to be built close to the road and not far from it. So the road is model 2, then the houses are likely to be in these points. And if the road is model 3, then the houses are likely to be around here. But let me put some numbers here. Let's say that this is your model, and this is a point x, and the point of line is over here. So you're going to generate a point close to the line using a Gaussian. So we put a Gaussian, and we center it at the point where the line meets the horizontal line. And we're going to sample out of that Gaussian. So we sample this one over here. So that's the same as saying, we're going to build a house that is likely to be close to the road. And it's going to be sampled from that vertical Gaussian over there. Now, if we have a bunch of points x1 up to x5, then we have five Gaussians, all centered at the point on the blue line. And from each one of these, we're going to sample some point. And those are the five points that we're sampling based on this blue line. And now all we have to do is find the line that best produce the points, the one that most likely produce these points. And coincidentally, that's going to be the same best fit of a line from linear regression. Why is this the case? Well, let's do some math. Let's say that the line has equation y equals mx plus b, and these are the points of intersection with the line. And these are the generated points. Now, these distances are going to be called d1, d2, d3, d4, and d5. And first, let's calculate the likelihood of generating this point. Since we have a Gaussian, then what really happens is that the Gaussian is centered at the point where x1 intersects that blue line. And we're generating this other point over here. Now, if we assume that it's a Gaussian with standard deviation one and mean equals zero, then the likelihood of generating this point is given by the formula of the Gaussian, which is one over square root of two pi e to the minus one half d1 squared. That is the likelihood of generating the first point. And we're looking at all of them. So that's the product of the five likelihoods. That's what we have to maximize. We have to maximize the likelihood of the line generating all those points. And since they're independent, then we have to look at the product. Now, a bunch of these things are constants, this one over square root of two pi. So let's forget about them. And let's maximize this over here. Now we can factor this as e to the minus a half times the sum of the squares of the d i's. And notice that if we maximize something, that's the same as maximizing the logarithm. So we can get rid of this e and also to the one half because that would just be multiplying by two. But let's remember that we have a negative number here. So that means we have to minimize the sum of squares. And minimizing the sum of squares is precisely the least square error. This is exactly what linear regression is. In linear regression, you want to find the blue line that minimizes the sum of square distances from the point. So this shows that finding the line that most likely produces a point using maximum likelihood is exactly the same as minimizing the least square error using linear regression. Here are three examples of lines and the Gaussians that they generate at each point. And as you can see, when we calculate the likelihoods for the actual data set of blue points, the winning one is the one in the middle, which we can see is the best line fit for the data set.