ALPS: An Application-Aware, Length-Predictive Scheduler for Efficient LLM Inference at Edge

Abstract

The deployment of Large Language Models (LLMs) on resource-constrained edge server clusters introduces a critical systems challenge: achieving high throughput and low latency under strict GPU memory and bandwidth limits. Current schedulers are ill-suited for this environment, as they treat requests independently, incurring two major inefficiencies. They waste compute and memory on redundant prefill computations for identical, lengthy system prompts across applications, and their phase-agnostic policies create suboptimal batches that poorly overlap compute-intensive prefills with memory-bound decodes, leading to GPU under-utilization. This paper presents ALPS, an application-aware, length-predictive scheduler for request serving for edge LLM inference. ALPS transforms the queueing layer into an intelligent orchestrator with two core techniques: (i) semantic-aware request grouping, which hashes requests by system prompt to co-locate them, reducing redundant prefill computations; and (ii) phase-aware scheduling, which uses a lightweight output token bucket predictor to estimate decode/prefill ratios and strategically reorders requests to maximize prefill-decode overlap in continuous hybrid batching. Augmented with SLO-aware preemption, ALPS is implemented as a practical scheduling layer atop vLLM. Extensive evaluations on an edge GPU testbed with diverse models and production workloads demonstrate that ALPS improves token throughput by up to 1.6× and reduces end-to-end latency by up to 2.4× compared to state-of-the-art schedulers.