Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
It is found that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth, not only on device bandwidth.
Joseph Kanichai, T. De Matteis, Animesh Trivedi
· 0 citations