Skip to content
Preprint

Simulate Before You Scale: High-Fidelity LLM Inference Planning Without GPUs

Oct 2026 · 0 citations · 49 references
Computer Science

TL;DR

BLIS is presented, a deterministic, CPU-only discrete-event simulator that models the full inference serving pipeline (admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption), and pairs analytical basis functions with learned scaling coefficients and overhead terms to predict step latency without per-model or per-GPU profiling.

Abstract

Deploying large language model (LLM) inference at scale requires jointly selecting hardware layout, parallelism strategy, scheduling policy, admission control, and request routing to meet latency and throughput targets. These choices interact: a routing decision that improves cache locality can concentrate load, and an admission threshold that triggers only at saturation allows queues to grow deep enough to degrade critical-tier traffic. Evaluating candidates on GPU clusters is too expensive for iterative exploration, yet existing predictive tools either ignore batching and queueing dynamics or model only fragments of the pipeline. In this paper, we present BLIS, a deterministic, CPU-only discrete-event simulator that models the full inference serving pipeline (admission, routing, scheduling, prefill/decode execution, KV-cache management, and preemption). BLIS pairs analytical basis functions with learned scaling coefficients and overhead terms, fit once globally, to predict step latency without per-model or per-GPU profiling. Across 36 experiments spanning six models and three GPU types, BLIS achieves 6.7% median end-to-end error while running around 200 times faster than real execution. In a configuration search over 420 serving configurations for Llama-2-70B, BLIS identifies a 4-GPU deployment that meets a 300ms mean TTFT service-level objective (SLO), whereas competing tools either require double the GPUs or recommend configurations that fail under load. Two policies evolved entirely in simulation were validated on real hardware and merged into llm-d, a widely adopted open-source platform for distributed LLM serving: an admission policy that reduces critical-tier TTFT P90 by up to 82%, and a flow-control policy that reduces E2E P50 latency by 32-48% on overloaded workloads.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput while meeting strict latency Service-Level Objectives (SLOs). Since state-of-the-art LLMs exceed the compute and memory capacity of a single GPU, inference is commonly dis...

Javad Mirzaei, Jeebak Mitra · 1 citation · ⚡1
Preprint Oct 2026

Compositional Performance Modeling of LLM Inference Clusters with Hierarchical KV Caching

Design-space exploration of LLM inference clusters requires predicting how hardware and serving policies interact with hierarchical key-value (KV) caching. Isolated operation profiles alone do not determine cluster latency: reuse changes computational demand, while KV location, capacity, and contention govern execution...

Rui-Yang Ma, Sheng Qi, Dai-Kang Kuang et al. · 0 citations
#machine learning Preprint Oct 2026

Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures

Large language model serving is increasingly limited not only by model size and GPU speed, but also by scheduling during autoregressive decoding, where continuous batching interleaves requests at token granularity as sequences arrive and finish. We study whether lightweight scheduling policies can improve latency-throu...

Ashman Mehra, Akshan Agrawal, Aryeh Carmi · 0 citations
#artificial intelligence Preprint Oct 2026

Stable Token-level Scheduling for Serverless LLM Inference

Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlo...

Hardi Ashishbhai Desai, K. BalajiN, Sourav Kanti Addya · 0 citations
Preprint Oct 2026

PDScope: Breaking the Prefill-Decode Seesaw for Memory-Bound MoE Inference

PDScope achieves 2.5-10.3x higher prefill throughput and 2.1-6.1x higher decode throughput than existing systems, showing that phase-aware co-design unlocks efficient MoE inference under memory constraints and extends naturally to multi-batch settings.

En-Da Yu, De-Zun Dong, Zhao-Ning Zhang et al. · 1 citation
Book Open access Sep 2026

MCSched: Memory-Controller-Aware Scheduling for Embodied LLM Workloads on NVIDIA Jetson

Preliminary but compelling evidence that MC-aware scheduling is a practical operating system/runtime direction for deployable embodied AI on unified memory edge devices is provided.

Teng Mei, Cheng-Xuan Pei, Marco Canini et al. · 2 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.