Sign in to continue learning
Structuring Machine Learning Projects
6h35m
In the last video, you saw how your dev and test set should come from the same distribution. But how large should they be? The guidelines for how to set up your dev and test sets are changing in the deep learning era. Let's take a look at some best practices. You might have heard of the rule of thumb in machine learning of taking all the data you have and using a 70-30 split into a train and test set. Or if you had to set up train, dev, and test sets, maybe you would use a 60% training and, say, 20% dev and 20% test. In earlier eras of machine learning, this was pretty reasonable, especially back when data set sizes were just smaller. So if you had 100 examples in total, these 70-30 or 60-20-20 rule of thumb would be pretty reasonable. Or if you had 1,000 examples, maybe if you had 10,000 examples, these heuristics are not unreasonable. But in the modern machine learning era, we're now used to working with much larger data set sizes. So let's say you have a million training examples. It might be quite reasonable to set up your data so that you have 98% in the training set and 1% dev and 1% test. I'm going to use D and T to abbreviate the dev and test sets. Because if you have a million examples, then 1% of that is 10,000 examples, and that might be plenty enough for a dev set or for a test set. So in the modern deep learning era, where sometimes we have much larger data sets, it's quite reasonable to use a much smaller than 20% or 30% of your data for a dev set or a test set. And because deep learning algorithms have such a huge hunger for data, I'm seeing that for problems where you have large data sets, that a much larger fraction of it goes into the training set. So how about the test set? Remember the purpose of your test set is that after you finish developing a system, the test set helps you evaluate how good your final system is. So the guideline is to set your test set to be big enough to give high confidence in the overall performance of your system. So unless you need to have a very accurate measure of how well your final system is performing, maybe you don't need millions and millions of examples in your test set. And maybe for your application, if you think that having 10,000 examples gives you enough confidence in the final performance, or maybe 100,000 or whatever it is, that might be enough. And this could be much less than, say, 30% of your overall data set, depending on how much data you have. For some applications, maybe you don't need a high confidence in the overall performance of your final system. Maybe all you need is a train and dev set. And I think not having a test set might be okay. In fact, what sometimes happened was people were talking about using train-test splits, but what they were actually doing was iterating on the test set. So rather than a test set, what they had was a train-dev split and no test set. If you're actually tuning to this set, to this dev set or this test set, it's better to call it the dev set. Although I think in the history of machine learning, not everyone has been completely clean and completely rigorous about calling it a dev set when it really should be treated as a dev set. But if all you care about is having some data to train on and having some data to tune to, and you're just going to ship the final system regardless and not worry too much about how it was actually doing, I think it would be healthier to just call it a train-dev set and acknowledge that you have no test set. This is a bit unusual. I'm definitely not recommending not having a test set when I'm building a system. I do find it reassuring to have a separate test set that you can use to get an unbiased estimate of how well it's doing before you ship it. But if you have a very large dev set so that you think you won't overfit the dev set too badly, maybe it's not totally unreasonable to just have a train-dev set. Although it's not what I usually recommend. So to summarize, in the era of big data, I think the old rule of thumb of a 70-30 split, that no longer applies. And the trend has been to use more data for training and less for dev and test, especially when you have very large data sets. And the rule of thumb is really to try to set the dev set big enough for its purpose, which is to help you evaluate different ideas and pick an algorithm A or B better. And the purpose of the test set is to help you evaluate your final classifiers. You just have to set your test set big enough for that purpose, and that could be much less than 30% of your data. So I hope that gives some guidance or some suggestions on how to set up your dev and test sets in the deep learning era. Next, it turns out that sometimes partway through a machine learning problem, you might want to change your evaluation metric or change your dev and test sets. Let's talk about when you might want to do that.