Skip to content

AgentKV: Phase-Aware KV Eviction for Agentic LLMs

Sep 2026 · 1 citation · 15 references
Computer Science

TL;DR

This work proposes AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union and implements AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online.

Abstract

Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV-eviction methods score cached keys against representative queries drawn from the most recent tokens, assuming future attention resembles recent attention. We show that agentic generation violates this assumption: future queries form a mixture over think, act, tool, and others phases, and principal-angle analysis shows these components occupy measurably different query subspaces, so recency representatives systematically undervalue keys that upcoming phases will need. We propose AGENTKV, which maintains a small query buffer per phase and scores cached keys against their union. We further implement AGENTKV in a persistent multi-turn serving path that carries compressed KV state across turns and compacts retained KV pages online. Across two models, six task domains, and three KV budgets each, AGENTKV improves task score by 5.5 points on average over R-KV and 5.3 over Tri-attention. Relative to upstream full-KV SGLang, AGENTKV improves output-token throughput by up to 1.80x. Code: https://github.com/LiuTaowen-Tony/agentkv.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management

This work proposes ActKV, the first KV cache compression framework tailored for agentic LLM inference, and establishes a compression criterion that values KV entries by their contribution to action generation and prioritizes action quality.

Zi-Han Wang, Cheng Tang, Lei Gong et al. · 0 citations
Preprint Oct 2026

Understanding KV Residency in Production-Scale Agentic LLM Workloads

Agentic large language model (LLM) applications turn one user request into a sequence of model and tool calls over a growing context. Retaining this context's key-value (KV) state in GPU memory lets later calls reuse it instead of recomputing it, but at fleet scale this needs substantial memory. We characterize four pr...

Mayank Arya, Agrim Bari, Anjaly Parayil et al. · 1 citation
#artificial intelligence Preprint Oct 2026

WAKMan: Workload Aware KVCache Manager

Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache eviction policy becomes a decisive factor in d...

Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, S. Varghese et al. · 1 citation
Preprint Sep 2026

ATTUNER: Recomputation-Free Agentic Context KV Reuse

This work isolates the source of the loss, finding that the positional mismatch has minor effect, and independently cached artifacts retain faithful representations: reading a provided artifact stays largely accurate, and performance degrades only when the model must select among multiple artifacts.

Xing-Hao Chen, Jun-Nan Dong, Cai Ke et al. · 1 citation · ⚡1
Review Open access Sep 2026

To Evict or to Merge? A Survey of LLM KV Cache Management Strategies and the SR-KV Framework

Large Language Models (LLMs) rely on a Key-Value (KV) Cache to avoid recomputation of attention information for the previously generated tokens, but the cache keeps growing linearly with the sequence length and thus quickly becomes the dominant memory bottleneck during long context inference. This paper presents a surv...

Chaitra Samant, Dhruv Panchal, Kevin Shah et al. · 0 citations
#artificial intelligence Preprint Oct 2026

AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning

AvoKV-E is introduced, a training-free eviction policy that first delays eligibility for recent states and then ranks eligible entries using candidate-normalized read pressure, key redundancy, and value-payload potential, which shows that long-reasoning KV eviction should preserve not only keys that are likely to be re...

Han Yu, Wen-Hui Zhu, Xi-Wen Chen et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.