Preprint

TACTICL: Task-Aware Compression of Tabular ICL Models

Mykhailo Koshil Matthias Feurer Katharina Eggensperger

Aug 2026 · 1 citation · 44 references

Computer Science

TL;DR

TACTICL is introduced, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning.

Abstract

The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression

arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction

This work adopts an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval in TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster.

Rasa Hosseinzadeh, Alex Labach, Zexin Xue et al. · 4 citations

Preprint Aug 2026

Tydra: An Efficient Hybrid Model for Tabular Data

Tydra is introduced, a hybrid Transformer-State Space Model (SSM) architecture for tabular in-context learning that interleaves attention and SSM layers that reduces inference time by 30% relative to TabPFN while retaining much of its predictive performance.

Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus et al. · 1 citation

Preprint Aug 2026

EXAONE Tabular 1.0 : Technical Report

These results establish EXAONE Tabular as a state-of-the-art compact tabular foundation model family, combining strong predictive performance across classification, point regression, and probabilistic regression with an efficient model design.

M. Eo, Min-Kook Suh, Hye-Seung Cho et al. · 12 citations · ⚡3

Preprint Aug 2026

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

Tabular foundation models (TFMs) achieve strong performance through in-context learning, but context-dependent inference imposes substantial latency and memory costs, hindering large-scale deployment. We propose GEAR (\emph{Generative Expansion and Real Anchoring}), a modular two-stage framework that distills TFMs into...

Qi Qin, Jia-Jie Zhu, Dali Chen et al. · 1 citation

Conference Open access Sep 2026

Representation-Aware Modularity: Efficient Cross-Task Generalization for LLMs

RaMod, a Representation-Aware Modularity framework, is proposed, a Representation-Aware Modularity framework to extend the ReFT paradigm to CTG through two novel components: Dual-Modular Representation & Parameter Fine-tuning, which manipulates only a strategically chosen subset of hidden representations with modular i...

Zheng Gong, Ying Sun, Chao Wang et al. · Proceedings of the Thirty-Fi... · 0 citations

Preprint Aug 2026

LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

This work argues that cross-modal alignment is implicitly captured in the information-compression trajectory, and proposes LLaVAFlow, an information-theoretic distillation framework that preserves alignment flow and enhances both downstream performance and generalization.

Mu-Yao Yuan, Mu-Yan Jiao, Jiang-Yong Ying et al. · 0 citations