This paper designs ACDC, a task-aware offline serving system that leverages inherent determinism and throughput orientation and designs a hybrid-serving mechanism that harvests these bubbles for offline tasks, improving end-to-end throughput by up to 35.7% over the state-of-the-art alternative.
Le-Ping Yang, Xue Li, Kun Qian et al.· Proceedings of the ACM SIGOP...· 0 citations
DPara is a PSD framework that reuses effective parallel drafters yet guarantees backbone--verification overlap in every round, thereby eliminating this probabilistic fallback altogether and surpassing the strongest serial and parallel speculative decoding baselines alike.
This paper shows that learned search can support arbitrary K with the preprocessing cost of a single top-1 model, and reduces total serving and preprocessing computation by up to 28%.
Yifan Peng, Jia-Fei Fan, Xing-Da Wei et al.· 0 citations
SMetric addresses session-centric scheduling with differential scheduling based on two indicators derived from the request itself, the session turn and the local KV\$ hit: it schedules first-turn requests for load balance, and sticks follow-ups to the instance with the highest local hit for high KV\$ reuse.