Sign in to continue learning
PyTorch: Techniques and Ecosystem Tools
33h1m
Welcome back. In this video, you'll dig deeper into the concept of optimization. At its core, optimization involves finding the best possible value for a function, and that's usually a maximum or a minimum. In machine learning, this often means tuning your model to achieve the best possible performance according to a metric like accuracy, precision, or recall. Let's revisit the app that you developed for the botanical garden in Course 1. The goal there was to identify flower types from images, and you aimed for high accuracy. Suppose that after building and training your model, you achieve an accuracy of 48%, meaning that the model correctly predicts the flower type in 48% of cases. Well, how can you improve on those results? One way is to adjust hyperparameters, which are settings that control how your model trains. One such setting is the learning rate, which determines the step size used by the optimizer during training. To understand how the learning rate affects accuracy, you can create a graph with learning rate values on the x-axis and the corresponding accuracy on the y-axis. Accuracy may be 48% at the current learning rate. If you were to test various learning rate values and determine the accuracy for each one, the chart might typically look like this. If the learning rate is too small, training can be slow, and the results may stagnate at suboptimal positions, resulting in low accuracy. If it's too large, the model might bounce around and never converge, also resulting in low accuracy. Somewhere in between is an optimal learning rate that maximizes accuracy. And this is the value that you should aim to find. Here's a chart that shows this exact pattern, taken from the lab that you'll complete right after this video. You can see that performance is low at both ends of the log scale, while it peaks in the middle, forming an inverted U-shaped curve. This illustrates that there's an optimal learning rate that gives the best performance. In short, the aim of optimization is to fine-tune hyperparameters, like the learning rate, to maximize outcomes such as accuracy, precision, recall, or F1 score. Or to minimize metrics, like training and inference time, mean squared error, or log loss. The learning rate is just one of many hyperparameters that we can tweak. And throughout this module, we'll explore several others. But before diving into model-specific enhancements, it's important to consider external factors that could be limiting performance. For example, do you have enough data? If your flower dataset lacks enough examples of a particular species, your model won't learn to recognize it well. By gathering more images, perhaps from open datasets, you can improve the model's performance. Also think about data quality. If labels are noisy, like a daffodil mislabeled as a sunflower, the model's learning will be compromised. Even the best model will be limited by the quality of its data. And as the saying goes: garbage in, garbage out. You should also evaluate feature representation. If your training images are noisy, blurry, or low-resolution, the model may struggle to find useful patterns. Getting higher-quality images or applying preprocessing techniques, such as resizing or cropping, can enhance visual feature distinction. Once your data is in good shape, you should focus on internal model improvements, which is the main goal of this module. This involves adjusting hyperparameters, including the model architecture design. Some examples include experimenting with different numbers of layers in your neural network, addressing overfitting by testing various dropout layers, experimenting with different batch sizes when training, and, of course, adjusting the learning rate. All of these adjustments, whether changing layer sizes, dropout rates, or optimizer settings, are central to model improvement. Throughout this module, you'll explore techniques to automate this process, making your optimization efforts even more efficient.