Conference
Sep 2026
Taichi: Elastic Multi-LLM Serving with Coordinated Memory–Compute Multiplexing
The rapid adoption of LLM applications is driving up serving cost, yet GPUs remain underutilized in practice due to single-model provisioning and highly variable, long-tailed workloads. Co-locating two or more LLMs on one GPU can improve utilization, but it requires fast and coordinated control of both KV cache/paramet...
Cun-Chi Lv, Xiao Shi, Zheng-Yu Lei et al.
· IEEE International Conferenc... · 0 citations