#artificial intelligence
Jun 2026
Forget Attention: Importance-Aware Attention Is All You Need
SSM-Informed Softmax Attention is proposed, which builds Mamba-3-style state transitions, rotation included, directly into the scores alongside FoX's gate and outperforms parameter-matched Transformer and FoX in language modeling and length extrapolation.
Suhyeong Shin, Yeongwook Yang
· arXiv.org · 0 citations