Skip to content
Preprint

Rethinking KV Cache Eviction for Agentic LLM Serving

Oct 2026 · 0 citations · 29 references
Computer Science

TL;DR

This work proposes preserving shared prefixes and evicting the KV cache from the beginning of the remaining history while retaining the cached suffix and builds Penelope around this strategy by co-designing the eviction policy, KV recomputation, and unified GPU-CPU cache management.

Abstract

Key-value (KV) caches impose substantial memory pressure on large language model serving systems, requiring cache eviction when capacity is insufficient. Existing mechanisms typically evict KV from the ends of cached sequences to preserve prefix sharing across requests. However, agent workloads repeatedly reuse long histories across successive turns. Recomputing the KV cache for tokens near the end of a sequence is substantially more expensive than for earlier tokens because their attention computation involves longer histories. To address this problem, we propose preserving shared prefixes and evicting the KV cache from the beginning of the remaining history while retaining the cached suffix. We build Penelope around this strategy by co-designing the eviction policy, KV recomputation, and unified GPU-CPU cache management. Returning requests reuse the KV cache retained in GPU or CPU memory and recompute only missing segments while preserving reuse of the cached suffix. We implement Penelope in SGLang and evaluate it on three Qwen3 model sizes using real agent traces. Compared with SGLang, Penelope reduces mean time to first token by up to 60% and job completion time by up to 37% with a GPU-only cache, and by up to 76% and 60% with a two-tier GPU-CPU cache.

View source

Similar papers

Sep 2026

The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems

KVSET, an online analyzer that estimates the KV cache working set of LLM serving workloads, substantially reduces the computational and memory overhead of conventional capacity-by-capacity simulation and makes online working-set analysis practical.

Lu-Chang Li, Shuai-Shuai Wang, Zhao Ruan et al. · 1 citation

When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse

This work studies production traces from two companies and evaluates 14 eviction algorithms across HBM-constrained and large memory-pool settings to show that effective prefix-cache management should retain recency as its foundation while selectively adding quick demotion for one-hit prefixes.

Yi-Yu Liu, Min-Lan Yu, Jun-Cheng Yang · 3 citations
#artificial intelligence Preprint Oct 2026

WAKMan: Workload Aware KVCache Manager

Prefix caching enables a Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation stage for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory resource, the KV cache eviction policy becomes a decisive factor in d...

Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, S. Varghese et al. · 1 citation
Sep 2026

Tessera: Demand-Driven KV Cache Management for Retrieval-Augmented LLM Serving

RAG and retrieval-based agent memory both inject retrieved content into LLM prompts, as document chunks and recalled memory records, respectively. The same content can recur across requests at different prompt positions or after different preceding contexts, preventing reuse through conventional prefix caching. Our cha...

Fei Fang, Chung-Hsiang Lo, Yi Liu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning

AvoKV-E is introduced, a training-free eviction policy that first delays eligibility for recent states and then ranks eligible entries using candidate-normalized read pressure, key redundancy, and value-payload potential, which shows that long-reasoning KV eviction should preserve not only keys that are likely to be re...

Han Yu, Wen-Hui Zhu, Xi-Wen Chen et al. · 0 citations
Preprint Sep 2026

ATTUNER: Recomputation-Free Agentic Context KV Reuse

This work isolates the source of the loss, finding that the positional mismatch has minor effect, and independently cached artifacts retain faithful representations: reading a provided artifact stays largely accurate, and performance degrades only when the model must select among multiple artifacts.

Xing-Hao Chen, Jun-Nan Dong, Cai Ke et al. · 1 citation · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.