MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
The surge of large language model (LLM) applications on personal devices imposes massive, bursty workloads on cloud serving infrastructure. While prefill-decode disaggregation improves throughput and scalability, memory-bound decode instances often suffer from persistent load imbalance, as output lengths are unknown wh...
Tian-Cheng Zhang, Yulin Chen, Yun-Feng Zhao et al.
· 0 citations