Sign in to continue learning
Sequences, Time Series and Prediction
21h36m
Once we have a model and a period, then we can evaluate the model on it, and we'll need a metric to calculate the performance. So let's start simply by calculating the errors, which is the difference between the forecasted values from our model and the actual values over the evaluation period. The most common metric to evaluate the forecasting performance of a model is the Mean Squared Error, or MSE, where we square the errors and then calculate their mean. Well, why would we square it? Well, the reason for this is to get rid of negative values. So for example, if our error was 2 above the value, then it would be 2. But if it were 2 below the value, then it would be minus 2. These errors could then effectively cancel each other out, which would be wrong because we have two errors and not none. But if we square the error value before analyzing, then both of these errors would square to 4, not canceling each other out and effectively being equal. And if we want the mean of our errors calculation to be of the same scale as the original errors, then we just get its square root, giving us a Root Mean Squared Error, or RMSE. Another common metric, and one of my favorites, is the Mean Absolute Error, or MAE. And it's also called the Mean Absolute Deviation, or MAD. And in this case, instead of squaring to get rid of negatives, it just uses their absolute value. This does not penalize large errors as much as the MSE does. Depending on your task, you may prefer the MAE or the MSE. For example, if large errors are potentially dangerous, and they cost you much more than smaller errors, then you may prefer the MSE. But if your gain or your loss is just proportional to the size of the error, then the MAE may be better. Also, you can measure the Mean Absolute Percentage Error, or MAPE. This is the mean ratio between the absolute error and the absolute value. This gives an idea of the size of the errors compared to the values. If we look at our data, we can measure the MAE using code like this. The Keras metrics libraries include an MAE that can be called like this. With the synthetic data we showed earlier, we're getting about 5.93. Let's consider that our baseline.