Verify Before You Fetch: Reliable KV Cache Compression for LLM Inference
Long-context applications, long-lived agents, and multi-agent systems require large language models (LLMs) to process and retain extensive information. In these settings, the KV cache can exceed GPU high-bandwidth memory (HBM) capacity, necessitating offloading to host memory, where CPU-to-GPU transfers can dominate in...
Devanshu Jha, G. Kumar, Agniva Chowdhury et al.
· 0 citations