Diagnosing Multi-Head Self-Attention: An Information-Theoretic Framework with Application to Time-Series Forecasting

Abstract

Background: Multi-head self-attention is central to Transformer-based time-series forecasting, yet its head-level information-selection behavior lacks a unified information-theoretic characterization. How much information a single head selects, how inter-head redundancy should be measured, and under what conditions a head can be removed without degrading predictions remain open questions. Methods: We treat each attention head as a discrete auxiliary selection channel whose conditional distribution is the attention weight vector. This yields a closed-form information identity and an entropy-dependent upper bound on selection information: I(X;Jth)≤logL−E[H(αth)]. We introduce total correlation—the Kullback–Leibler divergence between the joint head distribution and the product of its marginals—as a distributionally principled redundancy measure and relate head-removal sensitivity to conditional task information under population log-loss. Importantly, the selection-information bound characterizes input-dependent positional selection induced by attention weights, rather than the task information carried by the continuous value-weighted head output. Results: Synthetic experiments confirm the entropy-regularized optimality of softmax attention, the selection-information bound, and the redundancy decomposition under controlled conditions. Time-series forecasting experiments across nine benchmark datasets reveal that the head count achieving the lowest observed mean MSE varies across datasets, and that redundancy–sensitivity relationships are dataset- and head-count-dependent, though none remains statistically significant after multiple-comparison correction. Conclusions: The framework provides a principled diagnostic tool for analyzing selection behavior, inter-head dependence, and head-removal sensitivity in multi-head self-attention. It is a diagnostic framework rather than a new forecasting architecture or a standalone pruning algorithm. Pairwise redundancy carries diagnostic signal but is not, by itself, a complete predictor of head-removal sensitivity.