Er-Ci Xu

Batched in Back: Characterizing and Optimizing Offline LLM Inference in Production with ACDC

Serving offline large language model (LLM) inference workloads (e.g., log summarization and bulk translation) can consume up to 30% of GPUs in production. Despite this significant share, the characteristics of offline inference remain largely understudied. In this paper, we start by analyzing 1.5 million tasks comprisi...