This work proposes asking a different question: could a model's predictions result from any joint distribution?
Abstract
Tabular Foundation Models (TFMs) are currently the best approach to tabular prediction problems. They are constructed as transformers that approximate the Bayesian posterior predictive distribution based on a pre-training prior. These univariate predictors can be converted into multivariate ones autoregressively by sampling one target and adding it to the features. However, the faithfulness of the resulting joint has not been investigated. Furthermore, TFMs cannot be evaluated against the posterior itself, at least not on real-world datasets, because the ground-truth distribution is unknown. We therefore propose asking a different question: could a model's predictions result from any joint distribution? To answer this question, we pose two requirements that any such model must satisfy. The first is marginalization consistency, which demands that marginalized conditionals are equal to directly predicted marginals. The second is factorization consistency, which demands that different factorization orders result in equal joint distributions. Every TFM that we evaluate violates both of these requirements for both classification and regression across all datasets.
A task-centric, retrieval-based perspective is offered for how TFMs generalize: it is believed that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well.
Nour Shaheen, Junwei Ma, Alex Labach et al.· 1 citation
Existing benchmarks for OOD in tabular data are extended, providing evidence to support their adoption in high-stakes domains characterized by structural distribution shifts and identifying a scalability gap.
Malena Loza, David Chushig-Muzo, Eva Milara et al.· 0 citations
This review is, to the authors' knowledge, the first organised around the current generation of tabular foundation models, and taxonomises the architectures by pretraining regime, maps the capability space across five axes, isolates the language-model-on-tabular strand for prediction, feature engineering and generation, and summarises openness and deployment.
Sergei O. Kurashkin, V. Tynchenko, Alexey S. Borodulin et al.· Machine Learning and Knowled...· 0 citations
Pre-trained tabular prediction models based on Prior-Data Fitted Networks (PFNs), such as TabPFN and LimiX, have achieved remarkable progress in supervised learning, demonstrating immense potential across real-world scenarios and diverse downstream tasks. However, a critical question remains systematically unexplored: Does pre-training on data generated via causal mechanisms truly endow models with the ability to comprehend underlying causal structures? Furthermore, can these models leverage such causal information to achieve stable prediction across environments? To address these fundamental questions, we propose StablePFN, a novel tabular foundation model that integrates explicit causal awareness with stable predictive modeling. Leveraging a key yet largely overlooked advantage of the PFN paradigm, the availability of ground-truth causal structure during synthetic data generation, we train StablePFN to jointly identify the Markov Boundary (MB) of the target variable and perform the primary prediction task. We introduce an end-to-end ''Decouple-Discover-Predict'' architecture that utilizes sample decoupling weights to guide MB discovery, and employs a hard attention masking mechanism during inference to incorporate causal structural knowledge. Extensive experiments on both synthetic and real-world benchmarks demonstrate that StablePFN significantly outperforms state-of-the-art baselines in cross-environment prediction settings, particularly in challenging high-bias scenarios.
Zheng Guan, Yikang Chen, Hao Qian et al.· Proceedings of the 32nd ACM...· 0 citations
Modern AI models such as tabular foundation models and gradient-boosted ensembles can outpredict classical methods, but provide little basis for reasoning about their predictions. High-stakes decisions call for models that are both accurate and interpretable as built. Local linear modeling offers a path forward: a smooth regression function is locally well approximated by a linear one, allowing a linear fit near each query point to achieve high accuracy without sacrificing transparency. The challenges lie in learning what is"local"and developing statistical tools for interpretation. Here, we propose local distillation, in which a black-box"teacher"guides a regularized linear"student"model at each query point. The teacher (1) defines locality by upweighting training observations with similar predicted outcomes, and (2) anchors the fit with its prediction at the query point, included as a pseudo-observation whose weight is estimated from the data. For interpretation, we add a small amount of Gaussian randomization to the local objective and use refits to assess stability: selection frequencies identify reliable features at a query point, and clustering the randomized fits identifies stable subgroups across the data. Under the lasso penalty, we prove that this randomization yields feature-selection probabilities that are stable under small perturbations of the training responses. Across 17 benchmark datasets, local distillation nearly matches its AI teacher's accuracy while producing a sparse linear model at each test point. In a high-dimensional cancer gene expression example, the framework identifies patient subgroups whose local models use different genes; this heterogeneity is invisible to a global linear model, and difficult to surface in a black-box model.
A Bayes-filtered transformer (BFT) is a transformer trained on sequences that are generated in two steps: first a latent task is drawn from a prior, then observations are drawn conditional on that task. Trained under autoregressive log loss, the BFT's next-token prediction, in the idealized limit, is the Bayesian posterior predictive distribution (PPD) induced by that prior and that conditional law. In practice the trained BFT is only an approximation of this ideal PPD, raising an interpretive question: what prior and posterior over the latent task has the trained BFT actually internalized? Existing work answers this question by comparing the trained BFT's predictions against the predictions of various"reference"posteriors, each standing in for a different candidate algorithm or computation the BFT might be implementing. This prediction-space comparison is fragile: different posteriors can share the same posterior-mean predictions. We use predictive Monte Carlo (PMC) as a general interpretability tool for any BFT: using only next-token generation, PMC returns an approximation to the implicit prior and posterior over the latent task, answering the interpretive question directly in latent space. We apply PMC to three stylized task families spanning 0-Markov and 1-Markov exchangeability. The phenomena previously reported in these settings remain visible in latent space. Code is available at https://github.com/afiq-aswadi/bft-pmc
Afiq Abdillah Effiezal Aswadi, Haotong Ma, Susan Wei· 2 citations