Evolutionary Feature Construction for Tabular Transformers via Gradient-Enhanced Attention Rollout
Tabular Transformers learn feature interactions through attention, but they provide limited information about which interactions are most relevant for feature construction. Most attribution methods summarize influence on a single output token, leaving the token-to-token structure needed for feature construction hidden. We introduce GEAR, a gradient-enhanced attention rollout that preserves the full interaction matrix between input tokens and uses its input-input block as an interaction prior for a trained tabular Transformer. We use this prior to guide a validity-first genetic programming search, and make the search computationally feasible with an aligned frozen-backbone proxy that ranks many candidate cross features before retraining only the best few end-to-end. On the California Housing (CA), Microsoft (MI), and Jannis (JA) datasets, under the same 20-feature budget, GEAR improves over FT-Transformer and numerically matches or slightly exceeds OpenFE. In a contextual comparison across 11 datasets against published baselines, FT-Transformer + GEAR attains the best average rank.