Log In to continue learning
Tabular Foundation Models for Data Science Agents
Intermediate
1h32m
Now that you know how TabPFN learns from examples in context, you'll use it to make predictions yourself. You'll work on classification and regression problems, and see how TabPFN can give you both predictions, as well as estimates of the uncertainty. We'll also see how TabPFN can do time series forecasting by treating it as a tabular problem. All right, let's go. Let's start with classification, regression, and forecasting with TabPFN. You may have used common libraries like scikit-learn to train your own machine learning models or even use existing ones. And TabPFN is also scikit-learn compatible, It uses the same fit, predict terminology. However, what happens during fit and what happens during predict is slightly different. For classical machine learning models, when you fit a model, this usually means that you're training a model from scratch with your own dataset. And then predict is pretty fast because it just applies a prediction over the pre-trained model. However, with TabPFN, when we say .fit(), we're really just providing our training data or historical data as context to the model, but most of the work really happens during predict. We do the one forward pass over our context and the new rows that we want to predict, and we learn about our data, as well as predict during this predict call. Let's think about a use case where we have data about patients. This data comes in a tabular format and it holds information about patients, such as their age, but also, for example, information that may come from scans like chest pain, cholesterol levels, and so on. And we have a target value about whether a disease exists or not. All of this is historical data that we know about these patients. But what happens when we have a new patient and we don't yet know whether they have the disease or not? We want to be able to use both the information we now know about this new patient and all of the historical data as context to predict whether they would have the disease or not. We do this with no training. The model learns the historical table in context, and we have two options. Either we can use predict, which will give us the label that our model predicts directly, or we could use predict_proba, which would give us the probability distribution for all possible labels. In this case we have the option of label 0 and 1, so for example we might get the response that the probability of label 1 is 0.81. Once we've made predictions with our model, we can also have a look at how well the model did. For example, we can see what the predicted values of the labels were versus the actual values. So in our case, where we're looking at patients, we can have a look at how many we predicted to be sick versus how many actually were, and the same for healthy people. We can also use all of this information to evaluate our model and look at precision and recall. Precision in this case gives us: Of those flagged sick, what ratio is really sick? And recall tells us: Of the truly sick, what ratio did we catch? Other than predicting labels of specific classes, we can also use TabPFN to predict a number. We call this a regression task. Imagine a scenario where we have information about houses, like how big they are, how many bedrooms they have, what year they were built, etc., and we can use TabPFN to predict a specific price of what the sale price might be. However, we can also go beyond that. Every regression is also a distribution. We can use TabPFN to predict quantiles, for example, the 10th and 90th quantile, which basically gives us the edge of the estimation of what TabPFN thinks the value might be. You can think of these quantiles, 10 and 90 and everything in between, as the model saying that "I'm 80% sure that the value lies within this distribution." When these two numbers are close to each other, there's a higher confidence of what the actual value might be, but TabPFN may also provide a wider distribution, which should indicate to you that the model is not very certain about what the value would be. TabPFN can also do time-series forecasting. It turns out that if we represent time-series data as a tabular problem, we can use the TabPFN regressor as a way to predict future values as well. For example, let's think of a scenario where we have hourly electricity prices, and we want to predict what the next 48 hours of electricity might be. We can represent this information in tabular format and ask TabPFN to predict future price estimates. We can also add additional features to help TabPFN be a bit more certain about future predictions too. All right, let's see all of this in code. First, let's start by importing everything that we need. This includes a lot of helper functions to load various datasets if you want to try out different things yourself as well, and also the TabPFN client. For this lesson, we're going to be using the TabPFNClassifier and TabPFNRegressor via the TabPFN client. We also have this helper function called init_tabpfn, which takes care of everything for you so that you can use the client. We've already provided an API key so you don't have to worry about that. You do, however, still have the option of using TabPFN locally via our open source package. Let's start with a very simple classification task. We're going to try to predict whether a patient is at risk. We're going to use a dataset of 920 patients with some information about them. Let's start by looking at what the dataset looks like. We're going to load the heart disease dataset, and you'll see that we're using some common machine learning terminology here. We're splitting up our dataset into training sets and test sets. This is just going to allow us to evaluate how well our model did later on. You can however think of X_train and y_train as context features and context labels in the case of TabPFN. Let's have a look at the top three patients we have in our context features. Here, you see we have all of the information present in our data set about each patient, excluding the label as to whether they have the disease or not. For the same patients, we can also have a peek into what the labels are. For example, for patient 422, we've predicted the label 0, i.e., the disease does not exist. Now, let's have a look at what we have to do to initialize a TabPFNClassifier. first we simply initialize the classifier, but then we also need to fit the classifier. Here, we're fitting the classifier with X_train and y_train, which you can think of as your context features and context labels. Now that we have our classifier, let's try to predict the outcome for one patient that the model hasn't seen before. To do this, we're going to be using our test set, because we haven't fit the model on our test set. Let's have a look at one patient. This patient is 60 years old. We have some scans about the Blocked vessels, Thallium scans, et cetera. Now let's use our classifier to predict the outcome for this specific patient. We call classifier.predict over our patient, and then let's also print out the result. Because we're using the predict method here, we're getting the specific label that the model predicted. In this case, it predicted the presence of the disease, and we can also compare it to the test label, which is also one. So, in this case, the model has predicted a correct outcome. In some cases, instead of predicting the actual label, you may want to predict the probability for all available labels. In this case, we have two possible labels, and we use predict_proba to understand what the probability of label 0 is versus the probability of label 1. In this case, the model is pretty sure that the label is 1, i.e., the individual has the disease present. Note that you're using TabPFN via the client, and as models update, you may also get different results for these cells. At the beginning of the lesson, we kept 220 patients as a test set with their labels available to us, too. We can now run the classifier over all of these test patients and see how well our model is doing. In this confusion matrix, you can see that the model predicted 80 patients with no disease correctly, as well as 110 patients with the disease correctly, too. You can also see which ones it got wrong, where the disease should have existed, but it was predicted to not exist, and so on. Another thing you can do is look at some evaluation metrics like precision and recall we saw in the presentation. Now you've used a classifier, we can also use the TabPFNRegressor to predict numerical values. In this case, we're going to be predicting house prices. Let's start by loading our dataset and having a look at some examples. We have houses with the number of rooms they have, the year they're built, etc. We can also have a look at their sale prices. these are going to be our context features and context labels. So now for this numerical prediction task, let's have a new TabPFN, but this time let's use the TabPFNRegressor and fit it with this new data. Now, the same way we predicted the value of the label for one patient, let's try to predict the sale price of one specific house. In this case, we are selecting house at index 5. And we have some information about it, such as the number of rooms and the number of cars for the garage, year built, and so on. Now for this house, you can call regressor.predict, which by default will return to you the mean price. You can, however, get quantiles by specifying output_type="quantiles", which will give you any quantiles you ask for. So here we're asking for the 10th, 50th, and 90th percentiles. The single best prediction that the model returns is the mean, so in this case about $76,000, and we can compare that to the actual price, so there is about a thousand dollars discrepancy here. However, we also get a distribution from this prediction, Think of the percentiles as prediction edges. It starts at 10th percentile and goes up to the 90th percentile. So this model is pretty certain that the sale price lies between $61,000 and $90,000. The median is 77, but its mean prediction is 76, which is also its default prediction. Now we've run the prediction for one house, let's run it over all of our houses in the test set. Here, you can compare the predicted prices and the actual prices, in thousands, for six of the houses we have in our test set. The same way we evaluated how TabPFN classifies, we can also evaluate this regressor. We use the mean absolute error and the R-squared metrics. While mean absolute error tells us how far off the model is on average from the actual sales price, R-squared tells us how well the model is at explaining the price variation. 1 here is great, while 0 is no better than the model predicting the average value each time. Let's look at a few more plots for how well our model is doing. In this plot we are seeing how far off the model is at predicting from the actual price. The closer the blue dots are to the grey dashed line, the closer they are to the actual price. You'll notice how with lower prices, a) we have more examples and we're closer to the actual price, but with higher values, we have less instances, and we're further away from the actual sale price as well. We can also visualize the prediction band by asking for quantiles. In this case, the blue hue is the prediction band between the 10th and 90th percentiles, and you can see the orange dots are where we got it right, and the red dots are where the actual price lies outside of this band. Notice that the orange dots are where the actual price is within our prediction band, whereas the red dots are where the actual price lies outside our prediction band. Finally, let's try to use TabPFN to forecast time series. We have an electricity dataset for you to use, and we're going to try to predict the electricity price forecast for the next 48 hours. Here we have German electricity prices over hours, and we're going to use 336 hours as our historical context, and then we're going to try to predict the next 48 hours. Let's have a look at what our historical time series data looks like. With this data set, we get hour, price, load forecast, and renewable forecast as well. We are only going to use the hour and price, but later on, as a challenge to yourself, you can see how the prediction changes if you were to add additional features too. Let's see what happens when we add additional calendar features, like hour of day and day of week. Although our historical data has more columns, which you can try to use later on yourself, we're going to be focusing on the hour column, which also includes the date, and try to predict the price. To do this, we simply start by creating a TabPFNRegressor. Once we have the regressor, we fit it with our historical features, which is only hour in this case, and try and predict the price_eur column. Once we have a fitted model, we run predict over our future. Now, with this forecast price, let's plot out what it looks like, but let's also predict the quantiles so that we have a prediction distribution. You'll see here that the prediction distribution is relatively large, but the price forecast is still following the actual future pretty well. Now let's see what happens if we add some additional features to our dataset. Here, we're adding additional calendar features. We're using hour of day and day of week, which was essentially present in our hour column already. We have to now also refit the model with these additional features, and still, we want to predict price_eur. Now we visualize the same graph with these additional calendar features, and you'll notice that the probability band of the predictions are a lot narrower, so the model is a lot more certain about its predictions, and we're getting even closer to the actual price. As you see, providing TabPFN with additional features really helped with the time series forecasting here. Our dataset did include two additional features that we didn't actually use. So as a next challenge, you can have a look at the entire history dataset and see what happens if you add those features on top of the calendar features, and see if you can get this prediction band even closer to the actual price. In this lesson, you learned how to use TabPFN to do classifications, to predict numerical values, and even to do time series forecasting. In the next lesson, Elliot will walk through real-time inference.
