BLIS is presented, a deterministic, CPU-only discrete-event simulator that models the full inference serving pipeline (admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption), and pairs analytical basis functions with learned scaling coefficients and overhead terms to predict step...
Dipanwita Guhathakurta, M. Toslali, S. Parthasarathy et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.