SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Across reasoning, long-context understanding, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across attention budgets, with especially large gains under tight budgets, demonstrating more effective context ranking for downstream tasks.
Zhi-Wei Li, Lei Zhu, Hao Gu et al.
· 0 citations