Tian-Nan Fu

UNAS: Urgency- and Fairness-Aware Scheduling for SLO-Oriented LLM Serving

The widespread adoption and strong generalizability of large language models (LLMs) lead to highly heterogeneous workloads that exhibit substantial variability in request lengths and latency requirements. This pronounced heterogeneity causes existing scheduling strategies to suffer from head-of-line blocking and ineffi...