Skip to content

EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

Sep 2026 · 0 citations · 39 references
Computer Science

TL;DR

EfficientAgent is presented, which sizes and manages the host tier by this working set, which estimates the working set from agent histories to size the host tier and its predictions, made before the experiments, located the capacity at which offloading starts to pay.

Abstract

LLM agents resend their whole conversation on every turn, and most of it was already processed on the previous turn. Serving systems avoid recomputing it by caching its key-value (KV) state and, when GPU memory runs out, by offloading that state to host memory. For agents, offloading gives inconsistent results: on the same coding-agent workload it speeds up one deployment, slows down another, and changes nothing on a third, even where loading a token back is several times cheaper than recomputing it. The reason is that cached state must survive until it is used again. While one agent waits for its tool, the server processes the contexts of all other agents, so an agent's prefix is reused only if the host tier holds the reusable context of the whole agent pool, which we call the reuse working set. A smaller tier keeps writing state that is evicted before anyone reads it. We present EfficientAgent, which sizes and manages the host tier by this working set. A stack-distance model estimates the working set from agent histories to size the host tier; its predictions, made before the experiments, located the capacity at which offloading starts to pay. When the tier is too small, a runtime policy stops writing large refills of evicted context and keeps extending prefixes that are still cached; when the tier is large enough, it writes everything. On SWE-bench Verified coding agents, a host tier sized to the estimated working set cuts recomputed prompt tokens by 93% and end-to-end time by 39%. With a small fixed tier, the policy cuts recomputation by 35%; with a large tier, it avoids the 4.3-fold increase caused by always filtering writes. Across three GPU types and two models, offloading pays off when the GPU has little compute per byte of host bandwidth and the host tier holds the working set. Code is available at https://github.com/KunmingSHAO/efficientagent_release.

View source

Similar papers

Preprint Oct 2026

Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads

Agentic applications generate multi-turn sequences of LLM requests, with each new turn extending the context accumulated from prior interactions and tool calls. Prefix caching reuses the KV cache of earlier turns and avoids redundant prefill, but routing every turn to its cache-affine instance can create severe local q...

Lan-Cheng Zou, Bo-Wen Li, Wei Mao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

LazyAgent: Demand-Driven Materialization and Physical Optimization of Agentic Programs

LazyAgent is presented, a unified execution framework for agent-authored programs organized around a live, goal-derived demanded set that consistently outperforms the strongest goal-stopping eager baseline by refusing unrelated work before it starts and motivates benchmarks built from standing programs and sequences.

Xin Heng · 0 citations
Preprint Sep 2026

AKTS: Sub-Microsecond Kernel Policy Switching for Language-Model Agents

AKTS, which verifies a policy library once, at load time, and reduces the agent's runtime action to writing an integer index into an in-kernel array of preverified policies, and an in-kernel tail call resolves that index.

Mohammadali Khodabandehlou, Mahdi Alizadeh · 0 citations
#machine learning Preprint Oct 2026

Taking Tokenization off the Host for Agentic LLM Serving

GraphTok is presented, a GPU tokenizer that runs the entire encoding pipeline, from pre-tokenization through byte-pair encoding (BPE) merging to token-ID assembly, as one replayable CUDA graph.

Mohammad Siavashi, Pascal Schwinte, A. Banaei et al. · 2 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.