FlexKV: Resource-Efficient Remote KV Reuse for Low-Latency LLM Inference
An LLM inference system, FlexKV, is designed, which decreases TTFT while achieving high resource utilization by integrating two novel components: an adaptive Hybrid Adapter that chooses which reuse approaches to utilize, and a layer-wise Pipeline Scheduler that coordinates multiple compute and transfer operations.
Guo-Tao Yang, Xin-Pei Wang, Yi-Tao Hu et al.
· IEEE Transactions on Paralle... · 0 citations