Large Language Models without QK-Normalization
QK-MAP, a Magnitude-Aware Projection for constructing queries and keys, transforms the input representation using multiple learnable frequencies, so that changes in magnitude are encoded as bounded phase patterns rather than vector length alone.
Yong-Ding Tao, Yi-Tong Qiao, Tian-Qi Shao et al.
· 0 citations