AdaK: adaptive KV cache budget estimation framework for analyzing long-context large language model inference

Abstract

Introduction The deployment of LLMs on resource-constrained hardware is hindered by the memory-intensive KV Cache mechanism. Methods We propose AdaK, an adaptive KV cache budget estimation framework with three strategies: entropy-based thresholding, task-aware lookup table, and a lightweight policy network. Results AdaK reveals estimated KV cache reductions of up to 17.9% relative to fixed-k = 2048 baselines across 16 settings on Qwen3-4B, Qwen3-8B, and Mistral-7B. Discussion AdaK's decoupled design enables safe budget estimation as a dynamic ceiling for downstream sparse attention kernels.