FlexKV: Resource-Efficient Remote KV Reuse for Low-Latency LLM Inference
TL;DR
An LLM inference system, FlexKV, is designed, which decreases TTFT while achieving high resource utilization by integrating two novel components: an adaptive Hybrid Adapter that chooses which reuse approaches to utilize, and a layer-wise Pipeline Scheduler that coordinates multiple compute and transfer operations.
Abstract
Large language models (LLMs) have demonstrated exceptional performance across a variety of online services, which require real-time responses with low time-to-first-token (TTFT). To reduce TTFT, existing works commonly reuse the KV cache for requests sharing identical prefixes and leverage remote KV caches to increase the reuse rate. Existing remote KV-cache reuse policies, which we call resource-insensitive reuse policies, assume the network is stable and ignore real-time resource fluctuations. However, such policies cannot maintain low TTFT, as they neither make dynamic selections among different reuse representations nor leverage hybrid reuse to increase resource utilization, leading to significant waste of compute and transfer resources. We propose that an efficient LLM inference system should adopt a resource-efficient KV-cache reuse policy to reduce TTFT. To do so, we design an LLM inference system, FlexKV, which decreases TTFT while achieving high resource utilization by integrating two novel components: an adaptive Hybrid Adapter that chooses which reuse approaches to utilize, and a layer-wise Pipeline Scheduler that coordinates multiple compute and transfer operations. Evaluation across 14 unique model–workload combinations spanning 6.7B to 65B models, including a multi-node LLaMA-65B-GPTQ deployment, shows that FlexKV achieves up to a 4.90× TTFT speedup over evaluated deployable baselines. Compared with the analytical Optimal reference, FlexKV has a median TTFT gap of 8.10%.
