The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension
These results separate support geometry, which sets worst-case temperature scaling, from softmax-visible interaction geometry, which controls per-head approximation complexity, from softmax-visible interaction geometry, which controls per-head approximation complexity.
Yupei Sui, Jia-Ning Zhang
· 0 citations