This work proposes AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union and implements AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online.
Abstract
Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.
This work proposes ActKV, the first KV cache compression framework tailored for agentic LLM inference, and establishes a compression criterion that values KV entries by their contribution to action generation and prioritizes action quality.
Zi-Han Wang, Cheng Tang, Lei Gong et al.· 0 citations
Agentic large language model (LLM) applications turn one user request into a sequence of model and tool calls over a growing context. Retaining this context's key-value (KV) state in GPU memory lets later calls reuse it instead of recomputing it, but at fleet scale this needs substantial memory. We characterize four pr...
Mayank Arya, Agrim Bari, Anjaly Parayil et al.· 1 citation
Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache eviction policy becomes a decisive factor in d...
Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, S. Varghese et al.· 1 citation
This work isolates the source of the loss, finding that the positional mismatch has minor effect, and independently cached artifacts retain faithful representations: reading a provided artifact stays largely accurate, and performance degrades only when the model must select among multiple artifacts.
Xing-Hao Chen, Jun-Nan Dong, Cai Ke et al.· 1 citation· ⚡1
Large Language Models (LLMs) rely on a Key-Value (KV) Cache to avoid recomputation of attention information for the previously generated tokens, but the cache keeps growing linearly with the sequence length and thus quickly becomes the dominant memory bottleneck during long context inference. This paper presents a surv...
Chaitra Samant, Dhruv Panchal, Kevin Shah et al.· International Journal of Com...· 0 citations
AvoKV-E is introduced, a training-free eviction policy that first delays eligibility for recent states and then ranks eligible entries using candidate-normalized read pressure, key redundancy, and value-payload potential, which shows that long-reasoning KV eviction should preserve not only keys that are likely to be re...
Han Yu, Wen-Hui Zhu, Xi-Wen Chen et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.