Gaussian Equivalence for Multi-Head Self-Attention
Using random matrix theory, Gaussian equivalence is established for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output.
Tomohiro Hayase, Ryo Karakida
· 0 citations