High-Dimensional Learning Dynamics of Attention-Indexed Models
This work studies attention-indexed models, a broad framework that can represent multi-layer and multi-head attention architectures, and reveals that attention parameterization itself can act as an architectural implicit bias.
Yi-Zhou Xu, M. Sagitova, L. Zdeborová et al.
· 0 citations