The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
KVSET, an online analyzer that estimates the KV cache working set of LLM serving workloads, substantially reduces the computational and memory overhead of conventional capacity-by-capacity simulation and makes online working-set analysis practical.