Efficient LLM Coserving at the Edge via Resource-Aware Cooperative Scheduling

Abstract

Large language models (LLMs) are increasingly deployed on edge nodes to support edge intelligence applications. To overcome limited GPU memory, offloading-based methods partition model parameters between the GPU and host memory, enabling inference on commodity hardware. However, deploying a single model instance using the offloading-based method often results in significant infrastructure overhead and the underutilization of CPU, GPU, and PCIe resources due to a persistently idle CPU, bursty workload patterns, and bandwidth–compute mismatches. To address this issue, this article proposes RACS, a resource-aware cooperative scheduling (RACS) framework that enables a single edge node to coserve a latency-critical high-priority model and a latency-tolerant low-priority model. The key insight is that PCIe bandwidth constitutes the primary bottleneck in offloading-based inference. RACS comprises a runtime state manager that monitors PCIe availability in real time and a resource-aware cooperative scheduler that orchestrates the low-priority model accordingly. When the high-priority model is active, RACS restricts low-priority execution to preloaded feed-forward layers to avoid PCIe contention. When PCIe is idle, RACS aggressively utilizes GPU and PCIe resources while cooperatively scheduling computations on the CPU to maximize throughput. Extensive experiments with the OPT-13-B and OPT-6.7-B models under diverse prompt lengths, generation lengths, and real-world request traces demonstrate that RACS improves the throughput of offline tasks by up to 27.4% without compromising the latency of the high-priority model.