Log In to continue learning
Tabular Foundation Models for Data Science Agents
Intermediate
1h32m
Tabular data is behind many prediction problems, from healthcare and finance to business analytics. In this lesson, you'll learn what tabular foundation models are and how TabPFN uses in-context learning to make predictions without dataset-specific training, differently from traditional machine learning. Let's dive in. I would like to start from the basics. What is tabular data, and what are tabular prediction tasks? Here is a concrete example from medicine. You have 5,000 previous patients, one per row, and for each of them, you took a blood sample and measured four different omics blood markers, one per column. And you also know which of these patients had a particular disease, like in this case, early-stage Alzheimer's. We then measured these omics blood markers for new patients and would like to predict the probability that they have the disease to allow us to start the right treatment early. This is a typical tabular prediction task. You have training data where you have some measured features, Xtrain, and you know the value of the target variable, ytrain. And then you get the features of some new data, Xtest, and would like to make a probabilistic prediction for the target, ytest. This is a traditional formulation of supervised machine learning that the field has been using for over 50 years, and there's countless applications for this. You might, for example, like to predict: Will this treatment work for this patient? How strong will this material be? What type of galaxy is this data from? Which product will this customer buy? Will this credit be paid back, etc., etc.? There's literally thousands of examples. Let's make this breadth of applications concrete with a few more actual tables. So tabular foundation models can be used to predict how likely a new drug is to work based on the characteristics of a patient, like in this case. They can also be used to predict how likely a bank customer is to pay back a credit. And they can also be used to predict how much a customer is likely to spend in the next quarter, along with a full probability distribution. You might ask yourself if you can simply use LLMs for these tabular predictions. The answer is no, they're terrible for this task. And that is the case because they weren't built for this and structurally fail for tabular predictions. The reasons for this are plentiful. First, they don't understand entire numbers, but tokenize individual digits or pairs of digits and make stupid math mistakes like this. They're inherently sequential and therefore flatten tables into a very long sequence. If you have, for example, a million rows and a thousand columns, that's a billion numbers. And if you tokenize each of these, you have about three to four billion tokens. And all of these would have to go into the context window of your LLM. would simply break for this, and if somehow they could fit this in memory, then they would be millions of times too slow. LLMs also don't understand table invariances. If you exchange two rows or you exchange two columns, it's still the same data set, but the LLM doesn't realize that. LLMs are also much larger than they need to be. They have on the range of 10,000 times more parameters than the best tabular foundation models and are still way worse. You need that many parameters for encoding world knowledge, but you don't need it to encode statistical learning algorithms that learn from tabular data. Finally, LLMs are non-deterministic, and they hallucinate, both of which you really can't have in data science. So in summary, while we all love LLMs for unstructured data, they are far too slow, expensive, and inaccurate for tabular predictions. You can think of Tabular Foundation Models as the equivalent of LLMs, but actually made for tabular, structured data. Now, don't get me wrong. LLMs are amazing. They work on all kinds of unstructured data, no matter the domain. They excel at language and world knowledge. They're great at generating summaries and explanations, and especially coding, and thereby really boost our productivity. But the fact remains that they're terrible for predictions on structured data, like databases, CSVs, spreadsheets, and time series. to make predictions and forecasts on those, you really need to excel at numbers and statistics, and recognizing patterns. And if you can do that, you can directly plug into the machine learning value chain that has been set up for the last 50 years with traditional machine learning methods for tabular data. But the traditional workflow that data scientists use for tabular predictions doesn't scale, since each prediction task requires a lot of work. Given a dataset, the typical data science workflow usually includes cleaning the data and selecting the right machine learning model, engineering better features and tuning model hyperparameters, and finally, model deployment, maintenance, and governance. And given a new dataset, all of these steps repeat. Such data science projects often take three to six months until production, and a dedicated team for each problem. There's zero knowledge transfer between the models for each of these different tasks, and 60% of the machine learning engineering time is actually spent on model maintenance. You have these models flying around in the company that some intern built three years ago, and then the data starts changing and you need to update that model, and you have no idea how it was built. This severely limits the number of data science projects you can tackle. And like with language models, all of this operational inefficiency goes away when you use tabular foundation models. Instead of all these custom pipelines, you can simply get instant predictions in a single neural network forward pass. You simply input the data into a tabular foundation model, like TabPFN, and it makes predictions with no need for training or hyperparameter tuning. Some of our customers have replaced over a hundred custom models by a single tabular foundation model, TabPFN. And this can help data scientists become dramatically more efficient, so becoming a "10x data scientist". And this is no longer a vision for the future. Tabular foundation models have taken the lead in all kinds of tabular benchmarks by storm. In an area for which deep learning didn't used to work well. Gradient-boosted decision trees like XGBoost, LightGBM, and CatBoost ruled tabular data for over a decade. But then, over the course of two years, dozens of new tabular foundation models appeared, and they now completely dominate gradient-boosted decision trees, with a similar difference in Elo scores like there is between gradient-boosted decision trees and the simplest machine learning models out there like k-nearest neighbors. Now, how do tabular foundation models do this? Let's have a look at how we pretrain and how we apply them in practice. Let's start with the application. So we can simply download a pretrained TabPFN and apply it out of the box to arbitrary real-world datasets that the model never saw before. And without any training phase on this new dataset, you take the entire dataset, present it to TabPFN in-context, do a forward pass, and read out the probabilistic predictions for the test rows. This stage of applying TabPFN on new data is fast. It's really only a forward pass that does the predictions in context. On the left side is how we pre-train TabPFN. You don't really need to know this in order to apply it, just like you don't need to know how an LLM is trained in order to use it, but it's still helpful for understanding. So, let me show you. So, during pretraining, The types of inputs to TabPFN look like the same as you would see in practice: entire datasets with a question mark for the y_test portion. For the pretraining datasets, however, we know this y_test, and we compute a loss function that judges how likely this y_test is under the predictions that TabPFN makes. And this loss function is what we optimize. We use a standard stochastic gradient descent training that's used everywhere in deep learning in order to change the parameters of TabPFN to make the y_test more likely under the predictive distribution of TabPFN. And then we iterate. We throw away the dataset, generate another synthetic dataset from scratch, make a prediction on it, compute the loss, and change the weights of TabPFN again. And we do that hundreds of millions of times with a new dataset each time. Initially, the TabPFN parameters are randomly initialized, and the predictions are just noise. But with every dataset, it learns a little bit better to make predictions in context. It learns which pattern exists in the training portion of the dataset and how to apply these patterns to the test portion. And all of that is emergent behavior. We don't tell the network which parameters to update. It just figures out how to best use the dataset in context. And after enough training, TabPFN generalizes across datasets and can make strong predictions on entirely new datasets. And while previous machine-learning algorithms were handwritten, we have thus learned a new machine-learning algorithm that is executed by doing a single neural network forward pass. And we can do that, as we saw on the right, when we have a new real-world dataset to make strong predictions. So, in January 2025, we published the second version of this work, TabPFN v2, in Nature magazine, and that became a breakthrough moment for tabular data. In 2025, it was actually the most-cited paper published in Nature. In '24, that was AlphaFold, and in late 2025, DeepSeek-R1 was also published in Nature and is also highly cited by now. So, we're in really good company. TabPFN v2 was the first model that clearly outperformed gradient-boosted trees for tabular data, and became a standard tool for data scientists. Initially, TabPFN was limited to small datasets, with up to 10,000 rows only for TabPFN v2. But this limit quickly moved to 100,000 with TabPFN-2.5, a million with TabPFN-3, and then 10 million with TabPFN-4. So an increase of a factor of 1,000, going from 10,000 to 10 million, in less than two years. Next to having scaled up over the last years, TabPFN also has become ever easier to use. Like an open-weight language model, you can access it in a variety of different ways: locally downloading the weights from Hugging Face or by a simple pip install and running it on your machine, through our API where it runs on our GPUs, through a VPC where it runs in your own trusted cloud, or via agents talking to our MCP server. So I really encourage you to try it out. And many have indeed tried it. TabPFN has been used in thousands of prediction tasks in many different verticals like healthcare, finance, industrials, energy, tech, and many more. And in the next videos, you will actually use TabPFN yourself for both classification and regression.
