Using random matrix theory, Gaussian equivalence is established for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output.
Abstract
A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.
It is proved that a multi-head scaled dot product attention can be viewed as a parameter identification strategy, and a purely mathematical/statistical explanation, identification, for why specific architectural choices in transformers may have improved performance is discussed.
The framework provides a principled diagnostic tool for analyzing selection behavior, inter-head dependence, and head-removal sensitivity in multi-head self-attention and is a diagnostic framework rather than a new forecasting architecture or a standalone pruning algorithm.
Yan-Bin Zhang, Asif Ahmed Essak, Jiao Ding et al.· Information· 0 citations
Brain signal decoding is challenging because neural recordings are noisy and vary across individuals, while labeled data are often limited. Recent attention-based models on the symmetric positive definite (SPD) manifold have nevertheless achieved strong performance using covariance and connectivity representations, yet...
Standard Multi-Head Attention (MHA) computes contextual representations in a single forward pass, providing no mechanism to detect or correct diffuse or suboptimal attention distributions. Existing efforts to improve attention have primarily targeted computational efficiency or sequence length, leaving attention qualit...
R. Priyadarshini, Subham Divakar· Transactions on Artificial I...· 0 citations
Adaptive Anisotropic Attention is introduced, which splits attention into two paths: a temporal path, where each token attends to the tokens of its own electrode across time, and a spatial path, where it attends to the tokens of the other electrodes at the same time step.
Mahir Jain, P. Runwal, Aditya Mishra et al.· 0 citations
OBJECTIVE
Motor-imagery (MI) electroencephalography (EEG) offers a non-invasive route to brain-computer interaction, yet reliable single-trial decoding remains difficult because task-related patterns are weak, nonstationary, and distributed over multiple temporal scales. This study investigates whether a staged local-t...
Bin-Bin Chen, Chen Deng, Huan-Min Ge et al.· Journal of Neural Engineerin...· 0 citations
Learn how Google DeepMind and GovTech Singapore uses lightweight on-device Gemma models to intercept conversational scams in private chats with zero cloud data egress.
