This work establishes a theoretically sound and empirically proven approach for resolving memory-constrained scheduling in modern LLM deployments by introducing the Smallest Volume First (SVF) algorithm and its highly efficient variant, 1-bit SVF.
Lingjing Kong, Qi Qi, Yin-Yu Ye et al.· arXiv.org· 3 citations
Sorted-F, which repeatedly selects feasible batches using an F-metric that balances batch cardinality against downstream decode cost, achieves a constant-factor approximation guarantee in the unit-time, uninterrupted-decoding model with known output lengths.
Meixuan Wang, Yinyu Ye, Zijie Zhou· arXiv.org· 16 citations· ⚡2
A mathematical scheduling model that connects within-batch resource fairness to system throughput and provides a bi-criterion scheduling policy, ISJL, which maintains high throughput while aligning max-driven batch cost with token-metered revenue.
This work introduces a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards and applies an efficient bid-price control policy based on the online linear programming that admits requests when their SLO-weighted benefit exceeds their shadow pri...
Zixi Chen, Yinyu Ye, Zijie Zhou· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
