2026· Annual Meeting of the Association for Computational Linguistics· pp. 35311-35323· 0 citations· 27 references
Computer Science
TL;DR
O CTOPUS is proposed, a framework that confers fixed-memory inference onto pretrained Transform-ers without the information loss of linearization and outperforms state-of-the-art linearized baselines on the GSM8K benchmark, demonstrating that learned sparse retention serves as an effective regular-izer for long-horizon reasoning.
Abstract
Transformer inference becomes increasingly memory-bound as the Key–Value (KV) cache grows linearly with sequence length. While subquadratic architectures offer constant-memory inference, they rely on aggressive state compression that degrades performance on complex reasoning tasks. We propose O CTOPUS , a framework that confers fixed-memory inference onto pretrained Transform-ers without the information loss of linearization. O CTOPUS retrofits attention layers with Gated Selective Attention , a learnable module that enforces an adaptive sparsity policy over the context history. By dynamically scoring and retaining only high-utility KV states, this mechanism transforms the unbounded cache into a compact, evolving memory budget that filters out uninformative noise. Empirically, on the GSM8K benchmark, it outperforms state-of-the-art linearized baselines by over 36 points under identical memory constraints. Re-markably, O CTOPUS also surpasses its own full-cache teacher, demonstrating that learned sparse retention serves as an effective regular-izer for long-horizon reasoning.
Transformer-based large language models (LLMs) achieve strong performance across many tasks, but their Key-Value (KV) cache grows linearly with sequence length, creating a severe memory bottleneck for long-context inference. Existing heuristic eviction methods (e.g., H$_2$O and SnapKV) rely on static attention or positional signals that often fail to capture a token's future predictive influence. We propose DistillCache, a reinforcement learning framework that formulates KV-cache eviction as a sequential decision problem. DistillCache learns a lightweight policy network using rich internal model signals (attention statistics, value norms, entropy, and position) and trains it with REINFORCE via a per-step KL-divergence reward to preserve the full-cache output distribution. On a 7B-parameter instruction-tuned Transformer (Mistral-7B-Instruct-v0.3), DistillCache retains 94.2% of full-cache accuracy on LongBench at a 25% cache budget, outperforming both strong heuristic baselines (H$_2$O, SnapKV) by up to 2.7 absolute points and, under our re-implementations, concurrent RL-based methods (ForesightKV, RLKV) by up to 1.4 points on long-context tasks. On reasoning benchmarks, DistillCache is competitive with the best concurrent method and surpasses it under aggressive compression. It also delivers up to 2.1x full-cache throughput while maintaining competitive practical efficiency. These results highlight the effectiveness of learned, distribution-aware policies for memory-efficient long-context LLM inference.
MoNe is a lightweight modular neural memory that attaches to any frozen pretrained Transformer to enable long-context inference without retraining, achieving strong performance on needle-in-a-haystack and word extraction benchmarks from RULER, where ICL degrades sharply.
Won-Yong Cho, Kyubyung Chae, Tribhuvanesh Orekondy et al.· 0 citations
This work instantiates a new paradigm of incremental memory activation, where the effective capacity of memory is progressively expanded as the context grows, and applies this paradigm to state-of-the-art models, observing consistent improvements on standard language modeling and reasoning, as well as on long-context retrieval and understanding.
Reza Bayat, Ali Behrouz, V. Mirrokni et al.· 0 citations
DART (Decoded Attention over Recurrent sTates), which retains the chunk state contributions produced by the Mamba-2 chunked scan as chunk state memories, decodes token-conditioned keys and values from these memories, and performs state-memory attention (SMA) over the resulting KV pairs.
Yixiao Qian, Song Chen, Pengkai Wang et al.· 0 citations
MemDefrag, a training-free and model-agnostic framework that uses a middle-layer tracing signal to conduct memory defragmentation (rank, reorder, and filter memories), and applies an informativeness-guided proportional forgetting mechanism once capacity is exceeded, is proposed.
This work provides a new method for fine-tuning models with sparse attention that works for any KV cache policy, runs on a moderate hardware budget, and allows the model to co-adapt with the policy, often outperforming models trained with exact attention (sequence parallelism).
Matthias W. Seeger, Zeyu Zhang, Vihang Patil et al.· 0 citations