Sign in to continue learning
Machine Learning in Production
Intermediate
10h59m
Topics
Data Engineering
Deep Learning
MLOps
I think the use of HLP in machine learning had taken off partly because it helped people get papers published to show they can beat HLP. But it's also been a bit misused in settings where the goal is to build a valuable application, not just to publish a paper. When the ground truth is externally defined, then there are fewer problems with HLP when the ground truth really is some real ground truth. For example, I've done a lot of work on medical imaging, working on AI for diagnosing from x-rays or things like these. And given an x-ray image, if you want to predict a diagnosis, if the diagnosis is defined according to, say, a biopsy, so a biological or medical test, then HLP helps you measure how well does a doctor versus a learning algorithm predict the outcome of a biopsy or biological medical test. I find that to be really useful. But when the ground truth is defined by a human, maybe even a doctor labeling an x-ray image, then HLP is just measuring how well can one doctor predict another doctor's label versus how well can one learning algorithm predict another doctor's label. And that too is useful, but it's different than if you're measuring how well you versus a doctor are predicting some ground truth outcome from a medical biopsy. So to summarize, when the ground truth label is externally defined, such as the medical biopsy, then HLP gives an estimate for Bayes error and irreducible error in terms of predicting the outcome of that medical test, the biopsy. But there are also a lot of problems where the ground truth is just another human label. The visual inspection example we had from the previous video showed this, where the inspector had 66.7% accuracy. Rather than just aspiring to beat the human inspector, it may be more useful to see why the ground truth, which is just some other inspector compared to this inspector, don't agree. For example, if we look at the length of the different scratches that they label, say on these six examples, these were the lengths of the scratches. And if we speak with the inspectors and have them agree that 0.3 mm is the threshold above which a scratch becomes a defect, then what we would realize is that for the first example, both labeled at 1, totally appropriately. For the second example, the ground truth here is 1, but it's less than 0.3, so we really should change this to 0. Then 0.5 gets 1, 1, 0.2, 0, 0, 0.1, and this example has a scratch of 0.1, but really this should have been a 0. If we go through this exercise of getting the ground truth labeler and this inspector to agree, then we actually just raise human level performance from 66.7% to 100%, at least as measured on these six examples. But notice what we've done. By raising HLP to 100%, we've made it pretty much impossible for a learning algorithm to beat HLP. So that seems terrible. You can't tell the business owner anymore you beat HLP and thus they must use your system. But the benefit of this is you now have much cleaner, more consistent data, and that ultimately will allow your learning algorithm to do better. So when your goal is to come up with a learning algorithm that actually generates accurate predictions, rather than just proof for some reason that you can beat HLP, I find this approach of working to raise HLP to be more useful. To summarize, when the ground truth label Y comes from a human, HLP being quite a bit less than 100% may just indicate that the labeling instructions or labeling convention is ambiguous. On the last slide, you saw an example of this in visual inspection. You also see this in speech recognition where the comma versus ellipsis, dot, dot, dot, that type of ambiguous labeling convention will also cause HLP to be less than 100%. Improving label consistency will raise human level performance. And this makes it harder, unfortunately, for your learning algorithm to beat HLP, but more consistent labels will raise your machine learning algorithm performance, which is ultimately likely to benefit the actual application. So far we've been discussing HLP on unstructured data, but some of these issues apply to structured data as well. You already know that structured data problems are less likely to involve human labels, and thus HLP is less frequently used. But there are exceptions. You saw previously the user ID merging example, where you might have a human label where the two records belong to the same person. Or I've worked on projects where we would look at network traffic into a computer to try to figure out if the computer was hacked, and we asked human IT experts to provide labels for us. Sometimes it's hard to know if a transaction is fraudulent, and you just ask a human to label that. Or is this account a spam account, or a bot-generated account? Or from GPS, what is the mode of transportation? Is this person on foot, or on a bike, or in a car, or in a bus? It turns out buses stop at bus stops, and so you can actually kind of tell if someone's in a bus or in a car based on their GPS trace. And for problems like these, it would be quite reasonable to ask a human to label the data, at least on the first pass for a learning algorithm to make such predictions as these. And so when the ground truth label you're trying to predict comes from one human, the same questions of what does HLP mean? It is a useful baseline to figure out what is possible. But sometimes when measuring HLP, you realize that low HLP stems from inconsistent labels. And working to improve HLP by coming up with a more consistent labeling standard will both raise HLP and give you cleaner data with which to improve your learning algorithm's performance. Here's what I hope you take away from this video. HLP is useful and important for many applications. For problems where I think how well humans perform is a useful reference, I do measure HLP, and I use that to get a sense of what might be possible, and also use HLP to drive error analysis and prioritization. Having said that, if in the process of measuring HLP, you find that HLP is much less than perfect performance, much lower than 100%, then it's also worth asking yourself if that gap between HLP and 100% accuracy may be due to inconsistent labeling instructions. Because if that's the case, then improving labeling consistency will both raise HLP, but more importantly, help you get cleaner and more consistent labels which will improve your learning algorithm's performance.
