Log In to continue learning
PyTorch: Techniques and Ecosystem Tools
33h1m
it's important to first set a clear objective. Remember back in Course 1 when you worked on that botanical garden app? Your goal there was to develop a model that could accurately classify different types of flowers. To figure out how effective your model is. let's take a moment to revisit some common evaluation metrics in machine learning. starting with those that you became familiar with in Course 1. First. there's accuracy. And this measures the proportion of correct predictions made by a model. You calculate it by dividing the number of correct predictions by the total number of predictions. And this is a metric you typically want to maximize. The closer to one. the better. But be careful. Accuracy can be misleading when working with imbalanced datasets. where some categories are much more numerous than others. Because when one category dominates. your model can cheat by always predicting that category and still appear highly accurate. To better understand accuracy and the other metrics I'm about to mention. let's review the different types of classification outcomes and the errors that a model can make using an example from the botanical garden app. Imagine you're trying to identify whether an image is of a rose. If the image actually is a rose and your model predicts "rose". well. that's a true positive. If it's not a rose. but your model still predicts "rose". well. that's a false positive. And if it's not a rose and your model says so correctly. then that's a true negative. And finally. if it is a rose. but your model fails to recognize it. that's a false negative. This brings me to precision. which measures how often your model's positive predictions are actually correct. This metric can tell you. out of all of the times your app predicted rose. how many of them were actually roses. To calculate precision. you take the number of true positives and divide it by the positive predictions. which is the sum of true and false positives. The closer your precision score is to 1. the more confident you can be that your model's positive predictions are accurate. When does precision matter? It's especially important when false positives create significant issues. Say. for example. if you're implementing a plant disease detection feature in your app. If your model incorrectly predicts that a healthy plant has a disease. a false positive. you might recommend an expensive or invasive treatment that isn't needed. In these cases. precision helps ensure that when the model flags something as positive. it really means it. Next. you have recall. While precision asks: how many of my positive predictions were correct?. recall asks: how many of the actual positive cases did the model correctly identify? You calculate this by dividing the number of true positives by the sum of true positives and false negatives. and that is all actual positive cases. including the ones that the model missed. Your goal is to maximize recall to a value close to 1. Recall matters most when false negatives are costly. Going back to that app example. if your model misses a diseased plant. a false negative. the disease could spread and damage your entire crop. So in scenarios like this. you want high recall to make sure you're catching every important case. Often. you'll want to balance precision and recall because focusing too much on one can hurt the other. and that's where the F1 score comes in. It's the harmonic mean of precision and recall. giving you a single number that reflects both. Oh. yeah. and it's got nothing to do with Formula 1 racing. You want your F1 score to be as close to 1 as possible. It's especially useful when working with imbalanced data sets. where one class is more common than others. Fortunately. PyTorch makes calculating evaluation metrics easy with the Torchmetrics library. and this provides built-in classes for accuracy. precision. recall. and. of course. F1 score. In our example. we've created a function called evaluate_metrics() You can call the desired metrics and specify the classification task type as binary or multi-class. and then you specify the number of classes in your model. You can control how these metrics are averaged across classes using the average parameter. For example. macro will compute the metrics separately for each class and then average the results across all classes. This gives equal weights to all classes. regardless of how often they appear. And micro calculates the metric globally across all samples and classes. In the next lab. you're going to get hands-on experience using Torchmetrics to carry out these calculations. Now that you have a good grasp of common metrics to assess model performance. it's time to shift focus to the core of this module: optimization.