Evolutionary Feature Construction for Tabular Transformers via Gradient-Enhanced Attention Rollout

Abstract

Tabular Transformers learn feature interactions through attention, but they provide limited information about which interactions are most relevant for feature construction. Most attribution methods summarize influence on a single output token, leaving the token-to-token structure needed for feature construction hidden. We introduce GEAR, a gradient-enhanced attention rollout that preserves the full interaction matrix between input tokens and uses its input-input block as an interaction prior for a trained tabular Transformer. We use this prior to guide a validity-first genetic programming search, and make the search computationally feasible with an aligned frozen-backbone proxy that ranks many candidate cross features before retraining only the best few end-to-end. On the California Housing (CA), Microsoft (MI), and Jannis (JA) datasets, under the same 20-feature budget, GEAR improves over FT-Transformer and numerically matches or slightly exceeds OpenFE. In a contextual comparison across 11 datasets against published baselines, FT-Transformer + GEAR attains the best average rank.