Mixture of Semantic Attention Regimes (MoSAR) is introduced, which learns such an adaptive, controlled-decay geometry over query--key interactions, suggesting that it is not only adaptive, but also amenable to low-cost approximation at inference time.
Abstract
The quadratic complexity of dense self-attention remains a central bottleneck for long-context language modeling. Many efficient alternatives address this cost by deciding in advance where attention should be sparse or local. We argue that attention approximation should instead be approached as a geometric problem, with the relevant interaction geometry learned from data: natural-language dependencies are input-dependent and difficult to prescribe in advance, so the model should learn where positional relevance can decay and where broader interactions must be preserved. We introduce Mixture of Semantic Attention Regimes (MoSAR), which learns such an adaptive, controlled-decay geometry over query--key interactions. Input-conditioned query and key routers, applied after positional encoding, select mixtures over short, medium, and global regimes, inducing a continuous distance-dependent attention field rather than a fixed sparsity pattern. This geometry is learned during training and can subsequently be discretized through top-1 routing. In controlled pre-training experiments with matched 500M-parameter models, MoSAR learns a substantially lower-reach attention geometry without degrading language-modeling quality, improving perplexity over dense RoPE at the training context length. Under length extrapolation, MoSAR achieves the best perplexity among all evaluated variants, including strong baselines such as ALiBi. Moreover, the learned geometry remains stable under deterministic top-1 discretization, suggesting that it is not only adaptive, but also amenable to low-cost approximation at inference time.
Attention is a major scalability bottleneck in large language models (LLMs), with its computational cost and key-value (KV) memory traffic becoming increasingly prohibitive as context length grows. Sparse attention addresses this bottleneck by restricting each query to a subset of the context, but its effectiveness hin...
W. Raza, Ahsan Bilal, Yi-Dan Shen et al.· 0 citations
Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer...
Aleš Kubíček, Andrea Caciolai, Liang-Zu Peng et al.· 0 citations
Steering large language models typically relies on linear, context-independent interventions in activation space, an assumption that recent work has challenged and that can induce an information bottleneck when a fixed representation must encode many behavioral distinctions. We introduce MetaSteer, a method that learns...
Mehdi Jafari, Hao Xue, F. Salim· arXiv.org· 0 citations
The inference efficiency of Multimodal Large Language Models (MLLMs) is severely constrained by massive visual token sequences induced by high-resolution inputs, with computational cost scaling quadratically. Existing approaches primarily focus on downstream token compression, while overlooking a fundamental upstream i...
Zhi-Qiang Xia, Yang Li, Xin-Yuan Zhang et al.· arXiv.org· 0 citations
This work introduces Structured Matrix Attention via a family of causal masks with structured long-range routing whose row supports have VC-dimension, which makes VC-dimension an explicit knob governing access-pattern complexity, prefill cost, and decoding memory.
E. Anand, Abdullah Ateyeh, Archer Wang et al.· arXiv.org· 3 citations
Attention with Routed Memory is proposed a novel KV caching structure that introduces a fully differentiable, fixed-size memory system organized as a hierarchical router that achieves superior performance and efficiency compared to fixed KV-caching approaches, while remaining efficient and scalable in terms of both mem...
Qiu-Hao Zeng, Jerry M. Huang, Peng Lu et al.· arXiv.org· 0 citations