Sign in to continue learning
Machine Learning in Production
Intermediate
Topics
Data Engineering
Deep Learning
MLOps
For a lot of machine learning problems, the training sets and the depth and test set distributions start off being reasonably similar, but if you're using data augmentation, you're adding to specific parts of the training set, such as adding lots of data with cafe noise. So now your training set may come from a very different distribution than the depth set and the test set. Is this going to hurt your learning algorithm's performance? Usually the answer is no, with some caveats, when you're working on unstructured data problems. Let's take a deeper look at what that really means. If you're working on an unstructured data problem, and if your model is large, such as a neural network that is quite large and has large capacity and does low bias, and if the mapping from X to Y is clear, and by that I mean, given only the input X, humans can make accurate predictions. Then it turns out, adding accurately labeled data rarely hurts accuracy. This is an important observation, because adding data through data augmentation or collecting more of one type of data can really change your input data's distribution, the probability of X. Let's say at the start of your problem, 20% of your data had cafe noise, but using augmentation you added a lot of cafe noise, so now this is 50% of your data is data of cafe noise in the background. It turns out that so long as your model is sufficiently large, then it won't stop it from doing a good job on the cafe noise data, as well as doing a good job on non-cafe noise data. In contrast, if your model was small, then changing your input data distribution this way may cause it to spend too much of its resources modeling cafe noise settings, and this could hurt its performance on non-cafe noise data. But if your model is large enough, then this isn't really an issue. The second problem that could arise is if the mapping from X to Y is not clear, meaning given X, the true label of Y is very ambiguous. This doesn't really happen much in speech recognition, but let me illustrate this with an example from computer vision. This is very rare, so it's not something I would worry about for most practical problems, but let's see why this is important. One of the systems I had worked on many years ago used Google Street View images to read house numbers in order to more accurately geolocate buildings and houses in Google Maps. One of the things that system did was take as input pictures like this and figure out what is this digit. So clearly this is a 1, and this is an alphabet I. You don't see a lot of I's in Street View images, but there are some buildings. You may see a sign that says navigate to house number 42 I, but house numbers really rarely have an alphabet I in it. Now, if you find that your algorithm has very high accuracy on recognizing 1's, but low accuracy on recognizing I's, one thing you might do is add a lot more examples of I's into your training set. The problem, and this is a rare problem, is there are some images that are truly ambiguous. Is this a 1 or is this an I? And if you were to add a lot of new I's to your training set, especially ambiguous examples like this, then that may skew the data sets to have a lot more I's and hurt performance. Because we know that there are a lot more 1's than I's on house numbers, if the learning algorithm sees a picture like this, it would be safer to guess that this is a 1 rather than that this is an I. But if data augmentation skews the data set in the direction of having a lot more I's rather than a lot of 1's, then it may cause the algorithm to guess poorly on an ambiguous example like this. So this is one rare example where adding more data could hurt performance, and this example of 1 versus I is one that contradicts the second bullet because for some images, the mapping from X to Y is not clear. In particular, given only an image like this on the right, even a human can't really tell what this is. Just to be clear, the example that we just went through together is a pretty rare almost corner case, and it's quite unusual for data augmentation or adding more data to hurt the performance of your learning algorithm, so long as your model is big enough, maybe your neural network is big enough to learn from a diverse set of data sources. But I hope that understanding this rare case where it could hypothetically hurt gives you more comfort with using data augmentation or collecting more data to improve the performance of your algorithm, even if it causes your training set distribution to become different from your dev set and test set distribution. So far, our discussion has focused on unstructured data problems. How about structured data problems? It turns out there's a different set of techniques that's useful for structured data. Let's take a look at that in the next video.
