Sign in to continue learning
Okay, so this week you've been looking at machine learning for predicting the next value in a sequence. You've learned how to break your data down into window chunks that you could train on, and then you saw a simple single-layer neural network that gave you what was effectively a linear regression. Now let's take that to the next step with a DNN to see if we can improve our model accuracy. It's not that much different from the linear regression model that you saw earlier, and this is a relatively simple deep neural network that has three layers. So let's unpack it line by line. First, we'll have to get a data set which we'll generate by passing in our X-trained data along with the desired window size, batch size, and shuffle buffer size. We'll then define the model. Let's keep it simple with three layers of 10, 10, and 1 neurons. The input shape is the size of the window, and we'll activate each layer using a ReLU. We'll then compile the model as before with a mean squared error loss function and a stochastic gradient descent optimizer. Finally, we'll fit the model over 100 epochs, and after a few seconds training, we'll see results that look like this. It's pretty good still. And when we calculate the mean absolute error, we're lower than we were earlier, so it's a step in the right direction. But it's also somewhat of a stab in the dark, particularly with the optimizer function. Wouldn't it be nice if we could pick the optimal learning rate instead of the one that we chose? We might learn more efficiently and build a better model. Now let's look at a technique for that that uses callbacks that you use way back in the first course. So here's a code for the previous neural network, but I've added a callback to tweak the learning rate using a learning rate scheduler. You can see that code here. This will be called as the callback at the end of each epoch. What it will do is change the learning rate to a value based on the epoch number. So in epoch 1, it is 1 times 10 to the minus 8 times 10 to the power of 1 over 20. And by the time we reach the 100 epoch, it'll be 1 times 10 to the minus 8 times 10 to the power of 5, and that's 100 over 20. This will happen on each callback because we set it in the callbacks parameter of modeled.fit. After training with this, we can then plot the loss per epoch against the learning rate per epoch by using this code, and we'll see a chart like this. The y-axis shows us the loss for that epoch, and the x-axis shows us the learning rate. We can then try to pick the lowest point of the curve where it's still relatively stable like this, and that's right around 7 times 10 to the minus 6. So let's set that to be our learning rate, and then we'll retrain. So here's the same neural network code, and we've updated the learning rate, so we'll also train it for a bit longer. Let's check the results after training for 500 epochs. Here's the code to plot out the loss that was calculated during the training, and it will give us a chart like this, which upon first inspection looks like we were probably wasting our time training beyond maybe only 10 epochs, but it's somewhat skewed by the fact that the earlier losses were so high. If we crop them off and plot the loss for epochs after number 10 with code like this, then the chart will tell us a different story. We can see that the loss was continuing to decrease even after 500 epochs, and that shows that our network is learning very well indeed. And the results of the predictions overlaid against the originals looks like this, and the mean absolute error across the results is significantly lower than earlier. I'll take you through a screencast of this code in action in the next video. Using a very simple DNN, we've improved our results very nicely, but it's still just a DNN. There's no sequencing taken into account, and in a time series like this, the values that are immediately before a value are more likely to impact it than those further in the past. And that's the perfect setup to use RNNs like we had in the natural language course. You know, you'll look at that next week, but first let's dig into this code.