Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limi...
Hao Kang, Wei-Li Xu, Zi-Yang Li et al.· 0 citations
GPUs are expensive resources but often underutilized. This inefficiency is exacerbated by the increasing diversity of AI workloads, where small or specialized models often fail to saturate high-capacity devices, necessitating GPU sharing. Existing sharing approaches force a hard tradeoff. They either partition hardware...
Yi-Cheng Liu, Yi-Fan Qiao, Tian Xia et al.· Proceedings of the 32nd ACM...· 0 citations
This work presents EMA, a memory sharing system that allows GPUs within a server to borrow and reclaim memory from each other, forming an elastic pool of capacity, and ensures performance transparency for both borrowers and lenders.
Yi Xu, Tian Xia, Ion Stoica· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.