Zheng-Yu Lei

Taichi: Elastic Multi-LLM Serving with Coordinated Memory–Compute Multiplexing

The rapid adoption of LLM applications is driving up serving cost, yet GPUs remain underutilized in practice due to single-model provisioning and highly variable, long-tailed workloads. Co-locating two or more LLMs on one GPU can improve utilization, but it requires fast and coordinated control of both KV cache/paramet...