Queuing Generative AI Workloads with Variable Job Sizes and Memory Constraints
Generative AI inference presents a scheduling problem that differs from conventional compute workloads because requests vary substantially in service time and their memory requirements increase dynamically during autoregressive decoding. This study formulates generative AI serving as a memory-constrained, variable-size batch queue and develops a modified Pollaczek–Khinchine waiting-time approximation that includes a memory-contention penalty. It also proposes Memory-Aware Adaptive Batching (MAAB), an online admission-control policy that admits requests only when a probabilistic memory-headroom constraint is satisfied. MAAB combines projected peak memory demand with a size-aware priority score and an ageing mechanism to limit starvation. The framework was evaluated using discrete-event simulation with Poisson arrivals, shifted log-normal output lengths, and accelerator memory provisioned for heterogeneous request sizes. Four policies were compared: first-come-first-served, shortest-job-first, static batching, and MAAB. Across 50,000 completed requests and 20 replications, MAAB achieved a mean waiting time of 5.8 s, a 95th-percentile waiting time of 14.2 s, memory utilisation of 88.7%, three out-of-memory retries, and throughput of 58.6 jobs per minute. Relative to first-come-first-served scheduling, MAAB reduced mean waiting time by up to 68%, reduced out-of-memory-triggered retries by more than 95%, and increased throughput by 58%. These results support memory-aware, size-adaptive queueing as a practical approach to improving latency, utilisation, and memory safety in generative AI serving.