The proposed TabLoRA is a parameter-efficient trainable neural ensemble for large-scale tabular learning that improves the feasibility of neural ensemble learning while preserving much of the benefit of full ensembles.
Abstract
Tabular learning is still dominated by gradient-boosted decision trees (GBDTs), while recent deep learning approaches have become increasingly competitive. However, applying deep tabular models to large-scale datasets remains challenging, as large sample sizes, high feature dimensionality, or many target classes can introduce substantial computational cost. We propose TabLoRA, a parameter-efficient trainable neural ensemble for large-scale tabular learning. Instead of using fully independent ensemble backbones, TabLoRA shares a common backbone across predictors and introduces predictor-specific low-rank adaptations, enabling ensemble-style prediction without full parameter duplication. Across benchmarks, TabLoRA achieves a favorable balance between predictive performance and practical efficiency compared with GBDT methods and recent deep learning baselines under the same resource constraints. Memory analysis and ablation studies further show that the proposed design improves the feasibility of neural ensemble learning while preserving much of the benefit of full ensembles.
Experiments on the million-row HIGGS and SUSY datasets show that 512 prototypes retain strong predictive performance and reliable calibration, corresponding to an approximately 1,953-fold context compression.
Mahboobe Jadid, Melika Rezaye Garkani, A. Mousavi· 0 citations
TabNSM provides an effective and scalable approach to deep tabular regression, and demonstrates that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.
In deep learning for tabular data, efficient ensembles of multilayer perceptrons (MLPs) have recently emerged as effective and practical architectures. Existing methods of this kind use the same hyperparameters for all underlying MLPs, which requires hyperparameter tuning for achieving the best performance. In this work, we introduce TabPack, an efficient MLP ensemble with strong out-of-the-box performance and reduced reliance on traditional tuning. In a single run, TabPack samples and trains many MLPs with different hyperparameters efficiently in parallel and selects ensemble members on the fly during training. Thus, TabPack only requires specifying ranges from which to sample MLP hyperparameter rather than exact hyperparameter values, which naturally demands less precision for good performance. In experiments on medium-to-large public datasets, TabPack with default settings performs on par with extensively tuned prior methods, thus substantially reducing effort and compute resources needed to achieve competitive results on tabular tasks. Notably, running the default TabPack configuration on a modern MacBook took less time than tuning some baselines on an industry-grade GPU.
Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev et al.· 0 citations
Foundational models for tabular data have made significant progress in recent years, with TabICLv2 reporting state-of-the-art performance on several tabular classification tasks. However, full-context tabular ICL still suffers from attention cost that grows with the training-context size, which limits its ability to handle large datasets efficiently. Localized TabICLv2 introduces a method that reduces the inference cost of TabICLv2 by retrieving only the k nearest training neighbours for each test point, measured by similarity in the model's Stage 2 row-representation space, rather than using the full training context. This requires no architectural changes, and we show that accuracy retention can be improved through additional Stage 2 and Stage 3 fine-tuning. On TabArena classification tasks, the fine-tuned localized model retains 98.64% of Full TabICLv2 accuracy and it achieves a median 2.18$\times$ speedup in batch inference, and reaches approximately 249$\times$ median speedup in the single-query serving setting.
The advent of artificial intelligence has significantly enhanced credit‐scoring technologies. Despite the remarkable efficacy of advanced deep learning models, mainstream adoption continues to favor tree‐structured models due to their robust predictive performance on tabular data. Although pretrained models have seen considerable development, the use of such models for tabular‐structured credit‐scoring datasets remains largely unexplored. Tabular‐oriented large models, such as TabPFN, have made the application of large models in credit‐scoring feasible, albeit they can only process limited sample sizes. This paper proposes a novel framework that combines a tabular‐tailored dataset distillation technique with a pretrained model, thereby improving the scalability of TabPFN. Furthermore, although class imbalance is a common characteristic of financial datasets, its influence during dataset distillation has not been systematically explored; we therefore integrate imbalance‐aware objectives into the distillation process, resulting in improved performance on imbalanced credit data. Experiments on six publicly available credit‐scoring datasets show that the proposed imbalance‐aware distillation improves AUC by up to 8.7 percentage points over the standard MSE‐based distillation baseline. At the same time, distilled datasets retain roughly 76%–95% of the full‐data AUC while often using under 10% of the original training samples (and at most 31.3%) and consistently outperforming random subsets. Taken together, these results demonstrate that imbalance‐aware dataset distillation provides a compact yet accurate solution for imbalanced credit scoring and enhances the practical scalability of tabular pretrained models such as TabPFN.
Xia Li, Hanghang Zheng, Xiwei Zhuang et al.· Intelligent Systems in Accou...· 0 citations
With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.
Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al.· 0 citations
