TabFM is a model developed by Google Research for classification and regression tasks, introducing a different approach compared with traditional machine learning workflows on structured data.
It does not require training a new model on your own data. Instead, it uses in-context learning: historical data is provided as context, and TabFM uses those examples to generate predictions directly at inference time.
Conceptually, this is similar to the way language models use examples provided within a prompt.
Google has also integrated TabFM directly into BigQuery, where it can be used through SQL with AI.PREDICT and evaluated with AI.EVALUATE, without having to manage model training and deployment separately.
That naturally led us to a question:
how competitive can this approach be compared with the ML-based models already running within the ByTek Prediction Platform?
So we decided to run a few tests.
Analysis setup
We selected three real-world datasets with fairly different characteristics and compared TabFM with BPP models on classification tasks.
One important premise: this is not a definitive benchmark, nor is it intended to establish which technology is better overall.
It is an exploratory analysis based on three specific use cases that we found interesting enough to share.
The datasets:
Dataset A
Primarily demographic and channel-related features.
Positive rate: 6.5%
Dataset B
Richer behavioral features, including trials, demos and logins.
Positive rate: 11.8%
Dataset C
Demographic and transactional data.
Positive rate: 27.9%
To compare performance, we focused mainly on two metrics: ROC-AUC and PR-AUC.
ROC-AUC measures how well the model ranks the overall population, separating cases that are more likely to be positive from those that are more likely to be negative.
PR-AUC, on the other hand, focuses more closely on the model’s ability to identify positive cases correctly. It becomes particularly relevant when the positive class is less frequent and when, operationally, performance at the top of the ranking matters most.
Comparison results
On ROC-AUC, BPP models performed better across all three datasets.
On Dataset B, however, the difference was just 0.002 points, effectively making the result very close.
The picture is more nuanced for PR-AUC.
BPP models performed better on two datasets: by a wide margin on Dataset A and by a smaller margin on Dataset C.
TabFM performed better on Dataset B, the one with the richest behavioral signals.
The results, TabFM vs BPP models:
| Dataset | PR-AUC TabFM | PR-AUC BPP | ROC-AUC TabFM | ROC-AUC BPP |
| A | 0,286 | 0,542 | 0,845 | 0,910 |
| B | 0,771 | 0,713 | 0,939 | 0,941 |
| C | 0,816 | 0,903 | 0,915 | 0,952 |
Naturally, the most relevant metric depends on how the prediction is going to be used.
If the model is used, for example, to rank leads and decide which ones to contact first, the quality of the top of the ranking becomes critical, making PR-AUC particularly meaningful.
If, instead, the prediction score is used to build audiences, segment a population or exclude users with a very low probability of conversion, then the quality of the overall ranking becomes just as important, which makes ROC-AUC highly relevant.
Impact of increasing computational resources
On Dataset A, we wanted to understand whether increasing context size and computational workload significantly could help close the gap.
We moved:
- from 1,000 to 10,000 context rows;
- from 4 to 16 estimators.
In terms of workload, this represented roughly a 40x increase.
PR-AUC increased from 0.249 to 0.286.
The first increase produced an improvement of around 11% in relative terms. After that, each additional tripling of resources resulted in gains of roughly 1.8%.
In this case, therefore, increasing computational capacity was not enough to close the gap with the BPP model.
And this is where another part of the evaluation becomes relevant: predictive performance is only one part of the equation.
Architectural and operational limitations
During our tests with the TabFM library and pretrained weights, a number of limitations emerged that are worth considering.
- Available memory
In our local tests, we encountered out-of-memory errors before reaching some of the theoretically available library parameters, both on a Tesla T4 with 15 GB of memory and on an RTX 3080 with 20 GB, using a training dataset of roughly 45,000 rows.
This is related to the nature of in-context learning itself: the more historical data you provide as context, the more resources inference requires.
The managed BigQuery integration handles this differently, using sampling and distributed inference, but the underlying principle remains the same: the amount of context affects computational cost.
- Inference costs
With a traditionally trained model, the largest cost is usually concentrated in training and retraining. Once the model has been created, each new prediction does not require the entire training set to be reinterpreted.
With an in-context learning model, however, context is part of the inference process itself.
This changes the way costs need to be assessed, particularly when scoring needs to be repeated frequently or across large user populations.
In the BigQuery integration, TabFM usage is in fact associated with a pricing model linked to the amount of tokens processed.
- Model constraints
In the current BigQuery integration, classification problems support up to 10 classes and up to 20 features.
The library also does not provide native feature-importance tools comparable to those typically used in our workflows, meaning that additional interpretability analysis requires separate processing.
- Maturity and usage conditions
TabFM is still a relatively young project.
The code is open source, while the current pretrained weights are covered by a separate license that restricts commercial and production use. The repository also specifies that the library is not an officially supported Google product.
The BigQuery integration is currently available in Preview.
These are aspects that matter relatively little during experimentation, but become much more significant when evaluating adoption within production-grade predictive pipelines.
Final considerations
Based on these tests, we currently see no reason to replace BPP models.
In our experiments, they delivered the best PR-AUC results on two out of three datasets and the best ROC-AUC results on all three.
At the same time, the result on Dataset B is particularly interesting: with rich behavioral signals, TabFM not only came very close to our models on ROC-AUC, but also outperformed them on PR-AUC.
There is also a second scenario where this type of approach could become especially relevant: small datasets.
BPP models need to learn the structure of the problem from the data available to them. When the number of examples is limited, building a robust model inevitably becomes more difficult.
A foundation model, by contrast, starts from knowledge acquired during pre-training and uses new data as context.
This is probably one of the most interesting areas to explore further.
The goal is not necessarily to replace what already works, but to understand under which conditions a tabular foundation model can provide an advantage over established predictive models.
For now, we are not moving anything into production.
But we will keep testing TabFM and following its evolution.
Because the most useful question is probably not simply “foundation model or predictive models?”
It is understanding which approach to use, with which data, and for which type of decision.
