Skip to content
Book Open access

Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC

Sep 2026 · Proceedings of the ACM SIGOPS 32nd Symposium on Operating Systems Principles · 0 citations · 43 references

TL;DR

This paper designs ACDC, a task-aware offline serving system that leverages inherent determinism and throughput orientation and designs a hybrid-serving mechanism that harvests these bubbles for offline tasks, improving end-to-end throughput by up to 35.7% over the state-of-the-art alternative.

Abstract

Serving offline large language model (LLM) inference workloads (e.g., log summarization and bulk translation) can consume up to 30% of GPUs in production. Despite this significant share, the characteristics of offline inference remain largely understudied. In this paper, we start by analyzing 1.5 million tasks comprising 23 billion requests across text and multi-modal models. We discover that the key properties of offline workloads, namely inherent determinism and throughput orientation, are neither exploited by online LLM serving systems nor by existing offline serving frameworks. We design ACDC, a task-aware offline serving system that leverages these properties along three orthogonal axes, namely resource orchestration, cache management, and execution flow. ACDC has been deployed in production for over three months, achieving up to 3.9× higher output cost-efficiency. Beyond isolated deployment, we further identify that online prefill stages leave up to 38.3% of capacity idle as resource bubbles. We design a hybrid-serving mechanism that harvests these bubbles for offline tasks, improving end-to-end throughput by up to 35.7% over the state-of-the-art alternative.

Read PDF

Similar papers

Book Open access Sep 2026

LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems

LMTracer is presented, a fine-grained and real-time performance profiling framework for production LLM services that embed profiling logic into the execution through graph-embedded probing and streaming buffered profiling data to CPUs on demand to keep the execution of user kernels uninterrupted.

Wei Liu, Yong-Chao He, Bo-Han Zhao et al. · 0 citations
#edge computing Open access Dec 2026

Scaling LLMs via Collaboration: Efficient LLM Inference Across Resource-Constrained Devices

CoSEI is presented, a system that introduces operator-aware partitioning and synchronization-minimizing execution, guided by a lightweight planner and a tailored runtime, that demonstrates that with the right design, collaborative edge inference can make LLMs practical beyond the cloud.

Fei Zeng, Feng Lyu, Hao Wu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Stable Token-level Scheduling for Serverless LLM Inference

Serverless inference is emerging as an efficient deployment model for Large Language Models (LLMs) due to its ability to elastically scale resources, improve GPU utilization through resource sharing, and reduce infrastructure costs. However, existing approaches primarily optimize time to first token (TTFT) while overlo...

Hardi Ashishbhai Desai, K. BalajiN, Sourav Kanti Addya · 0 citations
Preprint Oct 2026

LLM Fine-Tuning Workloads: Characterization, Challenges, and Opportunities for Serverless AI

Serverless execution of Large Language Models (LLMs) is becoming important, reflecting their wide societal use. An increasingly present stage in LLM pipelines is fine-tuning, which specializes trained models and improves inference performance. Serverless operation of LLM fine-tuning workloads shifts operational concern...

Radu-Ionut Nicolae, Daniele Lotito, Alexandru Iosup · 0 citations
#artificial intelligence Preprint Oct 2026

Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs

Large Language Model (LLM) inference has become the dominant workload in modern AI systems, requiring serving infrastructures to maximize throughput while meeting strict latency Service-Level Objectives (SLOs). Since state-of-the-art LLMs exceed the compute and memory capacity of a single GPU, inference is commonly dis...

Javad Mirzaei, Jeebak Mitra · 1 citation · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.