Skip to content

StepKV: Step-Aware KV Cache Compression for LLM Agents

Aug 2026 · 0 citations · 39 references
Computer Science

TL;DR

Across multi-hop QA and long-horizon web reasoning tasks, StepKV sustains accuracy under low KV budgets where token-level baselines degrade sharply, offering a more robust efficiency-accuracy trade-off for multi-step agent inference.

Abstract

Key-value (KV) caching is essential for efficient autoregressive large language model (LLM) inference, but the cache grows linearly with context length, increasing storage and decoding costs. KV cache compression mitigates this cost by retaining only a subset of cached tokens. This challenge is particularly important for multi-step LLM agents, where a query expands into trajectories of reasoning, tool interactions, and retrieved observations. Existing pruning methods typically treat the cache as a flat token stream and rank tokens by recency or attention saliency. This creates a mismatch between the unit of compression and the unit of reasoning: token-level pruning removes individual entries, whereas useful information in multi-step agents is often organized into reasoning steps with uneven and delayed importance. Consequently, an early observation or intermediate decision may receive little recent attention yet remain essential for later evidence synthesis. We term this failure mode Reasoning Continuity Disruption.These observations motivate KV cache compression that jointly considers token- and reasoning-step-level information. StepKV addresses this goal by treating reasoning steps as first-class retention units. It associates cache entries with their generating steps, estimates step utility from trajectory-derived signals, and combines this utility with token-level saliency. The resulting scores globally rank prunable tokens, from which StepKV retains the top-scoring entries under a target budget. StepKV thus provides a step-centric perspective for agent KV cache compression. Across multi-hop QA and long-horizon web reasoning tasks, StepKV sustains accuracy under low KV budgets where token-level baselines degrade sharply, offering a more robust efficiency-accuracy trade-off for multi-step agent inference.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

iS-KV: Online Low-Rank KV Cache Compression via Block-Incremental SVD

Long chain-of-thought reasoning substantially increases KV-cache memory during autoregressive decoding, as every generated token introduces new key and value states and causes the cache to grow linearly with decoding length. Existing KV-cache compression methods typically control this growth through token eviction, but...

Yi-Ren Zhao, Guang-Hui Song, Tianrui Qin et al. · 0 citations
#natural language process... Conference Open access Oct 2026

Behavior-Preserving KV Cache Compression

This work proposes Behavior-Preserving KV Cache Compression, a training-free framework that scores candidate evictions by estimating the compressed-cache logits induced by their removal and evaluating the resulting KL to the full-cache next-token distribution using pre-eviction forward statistics.

Doo Hwan Hwang, Junyoung Jang, Jun-Ho Na et al. · 1 citation

CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

CompKV is introduced, the first compensation-aware sparse attention framework that divides tokens into blocks and explicitly optimizes selection for the downstream compensation mechanism, and shows that the residual left by block-level mean compensation is governed by both block attention mass and within-block logit va...

Zheng-Hong Huang, Rui-Zhe Yao, Dan-Yi Liu et al. · 1 citation · ⚡1
#artificial intelligence Preprint Sep 2026

ActKV: Efficient LLM Agents through Action-Guided KV Cache Management

This work proposes ActKV, the first KV cache compression framework tailored for agentic LLM inference, and establishes a compression criterion that values KV entries by their contribution to action generation and prioritizes action quality.

Zi-Han Wang, Cheng Tang, Lei Gong et al. · 0 citations
#artificial intelligence Preprint Sep 2026

ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction

Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. Reconstruction-based methods such as Attention Matching achieve strong downstream task p...

Zhe-Yu Shen, Guan-Hua Wang, De-Zhan Tu et al. · 1 citation

ARM: Attention with Routed-Memory for Learnable Sparse Control

Attention with Routed Memory is proposed a novel KV caching structure that introduces a fully differentiable, fixed-size memory system organized as a hierarchical router that achieves superior performance and efficiency compared to fixed KV-caching approaches, while remaining efficient and scalable in terms of both mem...

Qiu-Hao Zeng, Jerry M. Huang, Peng Lu et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.