TAO is presented, a tool-aware joint scheduling framework for LLM-based agentic serving that combines near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint; a dynamic program jointly selects and places waiting sessions under capacity constraints.
Shu-Hua Li, Sheng-Dong Ye, Dong-Dong Ge et al.· 1 citation
Sorted-F, which repeatedly selects feasible batches using an F-metric that balances batch cardinality against downstream decode cost, achieves a constant-factor approximation guarantee in the unit-time, uninterrupted-decoding model with known output lengths.
Meixuan Wang, Yinyu Ye, Zijie Zhou· arXiv.org· 16 citations· ⚡2
This work introduces a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards and applies an efficient bid-price control policy based on the online linear programming that admits requests when their SLO-weighted benefit exceeds their shadow pri...
Zixi Chen, Yinyu Ye, Zijie Zhou· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.