Preprint
Jul 2026
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Hierarchical Landmark Sparse Attention is proposed, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.
Xiang Hu, Xinyu Wei, Hao Gu et al.
· 3 citations