Taichi: Elastic Multi-LLM Serving with Coordinated Memory–Compute Multiplexing

Abstract

The rapid adoption of LLM applications is driving up serving cost, yet GPUs remain underutilized in practice due to single-model provisioning and highly variable, long-tailed workloads. Co-locating two or more LLMs on one GPU can improve utilization, but it requires fast and coordinated control of both KV cache/parameter memory and compute resources without disrupting in-flight requests. We present Taichi, a multi-LLM serving system for GPUlevel vertical scaling via joint memory-compute multiplexing. Taichi introduces (i) seamless memory sharing: a unified GPU memory pool with register-then-allocate management, Least-Common Alignment for heterogeneous KV layouts, and burst-time borrow/restore of idle parameter regions; and (ii) seamless compute switching: shared-memory-backed active-standby workers for millisecond-scale SM reallocation with bounded cross-model interference. Implemented atop vLLM and evaluated across single-GPU and distributed serving, Taichi reduces the bursting model's prefill P95 latency by 24-46× relative to S-Alloc's fixed 50/50 split across the four evaluated workload pairs, and reduces decode P95 latency by up to 2.9×, within the same GPU budget.