On-Demand Attention: Language Models Know When to Recall
Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, although the benefit of global access varies across prediction positions. We find that, before global attention is computed for the current step, the decoding states...
Hai-Bo Feng, Ruiqi Liang, Dong-Yang Jin et al.
· 0 citations