Online LLM serving is increasingly constrained by GPU memory capacity during decode. In length-skewed workloads, a small fraction of long requests keeps large key-value (KV) caches resident across many decode steps, exhausting GPU high-bandwidth memory (HBM), throttling admission, and reducing goodput for the short-req...
Dong Chen, Yu-Kun Zhou, Sheng Yao et al.· Proceedings of the 2026 ACM...· 0 citations
The booming LLM demand is driving a rapid increase in the scale and heterogeneity of modern AI infrastructure. To achieve high availability for sustained LLM workloads, infrastructure providers must contend with frequent and severe hardware errors. Without a holistic resilience understanding, providers may inadvertentl...
Xin-Kai Wang, Yun-Wei Li, Chao Li et al.· ACM Transactions on Architec...· 0 citations
Model-as-a-service platforms call for continuous optimization as complex serving conditions expose inefficiencies missed before deployment. Detailed always-on profiling can incur substantial overhead, while lightweight collection omits information needed for diagnosis. We present Herschel, a continuous optimization sys...
Lu-Ping Wang, Wei-Gao Chen, Yi-Fei Wu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.