Aug 2026· Asia-Pacific Workshop on Networking· 0 citations· 31 references
Computer Science
TL;DR
KV (Key-Value) volume is introduced, a physically grounded metric that captures the spatiotemporal footprint of a request’s KV cache occupancy, and it is shown that energy per KV volume (EPV) provides a stable and reproducible signature for modeling serving energy.
Abstract
Production large language model (LLM) serving uses continuous batching to maximize GPU utilization. While shared compute and memory resources improve throughput, they entangle the energy footprints of concurrent requests and make per-request energy physically unobservable. It is thus critical to develop a fair method for disaggregating device-level energy to individual requests. We tackle this challenge by developing a new approach to energy accounting in LLM serving. We introduce KV (Key-Value) volume, a physically grounded metric that captures the spatiotemporal footprint of a request’s KV cache occupancy, and show that energy per KV volume (EPV) provides a stable and reproducible signature for modeling serving energy. Building on EPV, we develop a state-aware energy model and a game-theoretic attribution method, and further build WattsOnLLM, a lightweight system for practical online energy accounting in existing LLM serving platforms with request-level granularity. We demonstrate that conventional heuristics deviate from the Shapley fairness target by 51.6% to 95%. In contrast, WattsOnLLM achieves a macroscopic energy estimation error as low as 1.9% and approximates the Shapley fairness target within 5.25% with very low overhead.
The results show that token attribution is not a reliable proxy for marginal energy under batched execution, and that measured Shapley ground truth can calibrate low-cost request features toward fairer attribution.
OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.
Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang et al.· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
GPU energy consumption represents a primary operational cost for Large Language Model (LLM) inference services. Despite the inherent variability of production workloads, most deployments rely on static power configurations that fail to exploit the non-linear relationship between power limits and performance. Consequently, opportunities to reduce energy overhead while maintaining Service Level Agreements (SLAs) are often missed. While modern GPUs provide hardware-level powercapping interfaces, their systematic application in a feedbackdriven manner to satisfy stringent latency guarantees remains significantly underexplored. This paper proposes a lightweight, non-intrusive adaptive GPU power-capping mechanism based on a PID controller that dynamically adjusts power limits using SLA feedback. The controller operates directly through vendor power-limit interfaces and incorporates guardrails such as deadbands, rate limiting, utilization-aware gating, and antiwindup protection. We evaluate the approach on a server with eight NVIDIA H200 GPUs running the qwen3:32b model under a time-varying Poisson workload, with SLA compliance defined over TTFT. Results show that static caps expose a rigid energy-SLA trade-off, whereas the adaptive controller achieves a 3.84% reduction in total energy consumption relative to a 700 W baseline while maintaining empirical SLA non-inferiority. These findings demonstrate that feedback-driven GPU power modulation can improve energy efficiency without compromising latency stability.
Alex F. R. Trajano, C. Costa, Francisco V. J. Nobre et al.· Annual International Compute...· 0 citations
Deploying large language models (LLMs) on edge nodes enables low-latency and privacy-preserving inference, but faces severe resource constraints under high-concurrence workloads. While existing inference systems leverage intranode key–value (KV) caching to improve efficiency, they largely neglect the unique complexities of multinode edge environments. Specifically, reactive KV cache eviction policies suffer from temporal uncertainty, often discarding reusable KV caches prematurely, while the tight coupling between request scheduling and cache placement often leads to myopic decisions that exacerbate load imbalance and resource contention. To address these challenges, we propose a dynamic block-level paradigm that treats KV blocks as the fundamental units for caching and scheduling, enabling dynamic sharing, generation, and eviction of arbitrary-length prefixes. We present complete modeling of the spatiotemporal coupling between scheduling and caching under block-level granularity, capturing intricate interactions overlooked by prior work. Based on this model, we design an online joint optimization algorithm, which applies to general edge LLM serving scenarios. The algorithm decouples spatiotemporal dependencies via randomized rounding over per-slot subproblems, achieving a balance between real-time responsiveness and long-term system efficiency. Theoretical analysis establishes high-probability near-optimality guarantees, and extensive experiments show that our method reduces the average time to first token (TTFT) by up to 54.02% over existing baselines.
Xishuo Li, Wei Jiao, Jun He et al.· IEEE Internet of Things Jour...· 0 citations
SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic, is introduced, which outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.
Pol G.Recasens, F. Agulló, Yue Zhu et al.· 0 citations