BLIS is presented, a deterministic, CPU-only discrete-event simulator that models the full inference serving pipeline (admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption), and pairs analytical basis functions with learned scaling coefficients and overhead terms to predict step...
Dipanwita Guhathakurta, M. Toslali, S. Parthasarathy et al.· 0 citations
Advances in quantum computing hardware and quantum algorithms are likely to cause major paradigm shifts in highperformance supercomputing environments. These shifts include foundational changes to system infrastructures that integrate both quantum and classical computational substrates through a combination of quantum...
M. Squillante, Asser N. Tantawi, Ming-Hung Chen· ACM SIGMETRICS Performance E...· 0 citations
Performance models of LLM servers support both latency evaluation and the design of controllers for autoscaling against service level objectives (SLOs) and for inference optimization. We model the multiplexed execution of prefill and decode operations with a tractable, approximate queueing model under Markovian assumpt...
Vishakha Ramani, Asser N. Tantawi· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.