BLIS is presented, a deterministic, CPU-only discrete-event simulator that models the full inference serving pipeline (admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption), and pairs analytical basis functions with learned scaling coefficients and overhead terms to predict step...
Dipanwita Guhathakurta, M. Toslali, S. Parthasarathy et al.· 0 citations
Each run of an AI-driven research system (ADRS) is an expensive search over a vast solution space, and dependable evaluation requires many runs, making run data both costly to produce and valuable to retain for large-scale analysis. Yet this data remains fragmented: teams operate in isolation, ADRS frameworks emit resu...
Leo Y. Lin, Vishakha Ramani, Z. B. Celik et al.· 0 citations
Performance models of LLM servers support both latency evaluation and the design of controllers for autoscaling against service level objectives (SLOs) and for inference optimization. We model the multiplexed execution of prefill and decode operations with a tractable, approximate queueing model under Markovian assumpt...
Vishakha Ramani, Asser N. Tantawi· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.