Skip to content

PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale

Sep 2026 · arXiv.org · Vol abs/2609.33224 · 0 citations · 45 references
Computer Science

TL;DR

PackServe is a scheduler designed to reduce resource costs while meeting latency SLOs for agentic LLM serving, which uses compact white-box models to predict latency under prefill/decode interference and packs requests onto fewer serving instances while preserving KVC reuse and SLO constraints.

Abstract

Request scheduling is a key challenge in large-scale clusters serving agentic large language model (LLM) workloads. An effective scheduler must preserve key-value cache (KVC) reuse across long, shared prefixes, meet token-level latency service-level objectives (SLOs), and minimize GPU resource footprint. Existing schedulers struggle to reconcile these requirements: request consolidation can sacrifice cache locality and increase prefill/decode interference, compromising both SLO attainment and resource efficiency. We present PackServe, a scheduler designed to reduce resource costs while meeting latency SLOs for agentic LLM serving. PackServe uses compact white-box models to predict latency under prefill/decode interference. Guided by these predictions, it packs requests onto fewer serving instances while preserving KVC reuse and SLO constraints, trading available latency headroom for improved per-instance throughput. Evaluation on 64 NVIDIA H20 GPUs shows that PackServe uses up to 16.8% and 24.6% fewer GPU-hours than state-of-the-art schedulers under 30-ms and 50-ms TPOT targets, respectively, while meeting the target TPOT objectives. PackServe has also been deployed in our production cluster comprising over 1000 GPUs, where it reduces the resource footprint by 34.7% compared with the original production scheduler.

View source

Similar papers

Preprint Aug 2026

TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving

A Task-Oriented Prefix-Aware Scheduler that jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution and scores candidate post-decision states by trading off the expected reduction in each task's longest remaining service path against the near-term benefit of downstream pre...

Hongqiu Ni, Han Tian, Chi Zhang et al. · 3 citations
Open access Aug 2026

CELLServe: An SLO-Aware and Cost Efficient LLMs Serving System for Serverless Computing Environments

CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.

Ze-Jian Wang, Nan Lin, Zi-Nuo Cai et al. · 0 citations
Book Open access Sep 2026

Heterogeneous SLO Guaranteed Multi-Resource-Aware Batching in LLM Serving

In this paper, we study a mixed-prompt scenario—where both short and long prompts coexist—in an LLM inference serving system that supports diverse applications with heterogeneous iteration-time SLOs. To improve throughput for long prompts, prior work divides them into chunks and batches requests or chunks to meet the t...

Hai-Ying Shen, Tanmoy Sen, Yuxiong He · 0 citations

Mix-or-Split: Latency-Aware Scheduling for Edge–Cloud LLM Inference

In this paper, we present a latency-aware scheduler for large-language-model (LLM) inference across mobile devices, edge servers, and a remote cloud. Our fine-grained delay model captures OFDMA uplink/downlink rates, KV-cache backhaul serialization, and profiled GPU planning-chunk resource constraints, enabling per-req...

Xinghan Wang, Xiao-Xiong Zhong, Wei-Hong Yang et al. · 0 citations
Book Open access Sep 2026

CrossServe: Cross-Layer Scheduling for SLO Optimization in Multi-Tenant LLM Serving

CrossServe is introduced, a cross-layer scheduling framework that jointly optimizes request routing, adaptive micro-batching, and tenant-weighted preemption to maximize SLO attainment in multi-tenant LLM serving and increases overall SLO attainment by 53, 29, and 14 percentage points respectively.

Jia-He Li, Jia-Bin Li · 0 citations
Preprint Aug 2026

CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving

When affinity recovers too little KV work, its residual load skew reduces or erases the improvement, so gating any deployment with a shadow replay rather than enabling affinity from workload statistics alone is recommended.

Huang Cheng · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.