BLIS is presented, a deterministic, CPU-only discrete-event simulator that models the full inference serving pipeline (admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption), and pairs analytical basis functions with learned scaling coefficients and overhead terms to predict step...
Dipanwita Guhathakurta, M. Toslali, S. Parthasarathy et al.· 0 citations
Prefill-decode disaggregation is becoming a common architecture for LLM serving because it separates two phases with distinct execution patterns and SLO objectives. Existing systems typically combine a fixed prefill/decode worker ratio with request routing across workers. However, real-world workloads exhibit both shor...
Kartik Ramesh, Kai-Di Fu, Zi-Han Zheng et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.