Sign in to continue learning
Advanced Learning Algorithms
34h24m
You've seen how by looking at Jtrain and Jcv, that is the training error and the cross-validation error, or maybe even plotting a learning curve, you can try to get a sense of whether your learning algorithm has high bias or high variance. This is a procedure I routinely do when I'm training a learning algorithm. I'll often look at the training error and the cross-validation error to try to decide if my algorithm has high bias or high variance. It turns out this will help you make better decisions about what to try next in order to improve the performance of your learning algorithm. Let's look at an example. This is actually the example that you had seen earlier. If you've implemented regularized linear regression on predicting housing prices, but your algorithm makes unacceptably large errors in its predictions, what do you try next? And these are the six ideas that we had when we had looked over this slide earlier. We looked at more training examples, tried a smaller set of features, additional features, and so on. It turns out that each of these six items either helps fix a high variance or a high bias problem. And in particular, if your learning algorithm has high bias, three of these techniques will be useful. If your learning algorithm has high variance, then a different three of these techniques will be useful. Let's see if we can figure out which is which. First, one is get more training examples. We saw in the last video that if your algorithm has high bias, then if the only thing we do is get more training data, that by itself probably won't help that much. But in contrast, if your algorithm had high variance, say it was overfitting to a very small training set, then getting more training examples will help a lot. So this first option of getting more training examples helps to fix a high variance problem. How about the other five? Do you think you can figure out which of the remaining five fix high bias or high variance problems? I'm going to go through the rest of them in this video in a minute, but if you want, be free to pause the video and see if you can think through these five other things by yourself. So feel free to pause the video and... Just kidding, that was me pausing, not your video pausing. But seriously, if you want, go ahead and pause the video and think through it if you want or not, and we'll go over these with you in a minute. How about trying a smaller set of features? Sometimes if your learning algorithm has too many features, then it gives the algorithm too much flexibility to fit very complicated models. This is a little bit like if you had x, x squared, x cubed, x to the fourth, x to the fifth, and so on. And if only you were to eliminate a few of these, then your model won't be so complex and won't have such high variance. So if you suspect that your algorithm has a lot of features that are not actually relevant or helpful to predicting a housing price, or if you suspect that you had even somewhat redundant features, then eliminating or reducing the number of features will help reduce the flexibility of your algorithm to overfit the data. And so this is a tactic that will help you to fix high variance. Conversely, getting additional features, that's just adding additional features, is kind of the opposite of going to a smaller set of features. This will help you to fix a high bias problem. That's a concrete example if you were trying to predict the price of a house just based on the size. But it turns out that the price of a house also really depends on the number of bedrooms and on the number of floors and on the age of a house. Then the algorithm will never do that well unless you add in those additional features. So that's a high bias problem because you just can't do that well on the training set when you know only the size. It's only when you tell the algorithm how many bedrooms are there, how many floors are there, what's the age of the house, that it finally has enough information to even do better on the training set. And so adding additional features is a way to fix a high bias problem. Adding polynomial features is a little bit like adding additional features. So if your linear function, straight line, can't fit the training set that well, then adding additional polynomial features could help you do better on the training set. And helping you do better on the training set is a way to fix a high bias problem. And then decreasing lambda means to use a lower value for the regularization parameter. And that means we're going to pay less attention to this term and pay more attention to this term to try to do better on the training set. And again, that helps you to fix a high bias problem. And finally, increasing lambda, well, that's the opposite of this. But that says you're overfitting the data. And so increasing lambda would make sense if it's overfitting the training set, just paying too much attention to fitting the training set, but at the expense of generalizing to new examples. And so increasing lambda would force the algorithm to fit a smoother function, maybe a less wiggly function, and use this to fix a high variance problem. So I realize that this was a lot of stuff on this slide, but the takeaways I hope you have are, if you find that your algorithm has high variance, then the two main ways to fix that are either get more training data or simplify your model. And by simplify model, I mean either get a smaller set of features or increase the regularization parameter lambda. So your algorithm has less flexibility to fit very complex, very wiggly curves. Conversely, if your algorithm has high bias, then that means it's not doing well even on the training set. So if that's the case, the main fixes are to make your model more powerful, to give it more flexibility to fit more complex or more wiggly functions. And so some ways to do that are to give it additional features or add these polynomial features or to decrease the regularization parameter lambda. By the way, in case you're wondering if you should fix high bias by reducing the training set size, that doesn't actually help. If you reduce the training set size, you will fit the training set better, but that tends to worsen your cross-validation error and the performance of your learning algorithm. So don't randomly throw away training examples just to try to fix a high bias problem. One of my PhD students from Stanford, many years after he'd already graduated from Stanford, once said to me that while he was studying at Stanford, he learned about bias-invariance and felt like he got it, he understood it. But that subsequently, after many years of work experience in a few different companies, he realized that bias-invariance is one of those concepts that takes a short time to learn but takes a lifetime to master. Those were his exact words. I think bias-invariance is one of those very powerful ideas. When I'm training learning algorithms, I almost always try to figure out if it is high bias or high variance. But the way you go about addressing it systematically is something that I think you will keep on getting better at through repeated practice. But you'll find, I think, that understanding these ideas will help you be much more effective at how you decide what to try next when developing a learning algorithm. Now, I know that we did go through a lot in this video. And if you feel like, boy, this is a lot of stuff here, it's okay, don't worry about it. Later this week, in the practice labs and practice quizzes, we'll have also additional opportunities to go over these ideas so that you can get additional practice with thinking about bias-invariance of different learning algorithms. So if it seems like a lot right now, it's okay. You get to practice these ideas later this week and hopefully deepen your understanding of them at that time. Before moving on, bias-invariance also are very useful when thinking about how to train a neural network. So in the next video, let's take a look at these concepts applied to neural network training. Let's go on to the next video.