TL;DR
A mathematical model is proposed to predict throughput and energy consumption for concurrently executing CV workloads on edge GPU accelerators and can be integrated into functional simulation frameworks for edge–cloud deployment studies.
The rapid evolution of generative artificial intelligence has substantially increased the computational demands of image synthesis models, traditionally restricting their execution to centralized cloud infrastructures. In response to concerns related to data privacy, energy consumption, cost, and dependency on hyperscale providers, this work investigates the feasibility of executing state-of-the-art generative image models at the network. We present a quantitative performance evaluation of two representative model families, Stable Diffusion XL (SDXL) and Z-Image Turbo, executed on heterogeneous hardware, including high-end consumer GPUs, mobile-class devices, and legacy workstation GPUs from NVIDIA and AMD. The analysis focuses on latency, power consumption, resource utilization, and the impact of software stack optimizations, such as attention mechanisms and backend frameworks, under realistic hardware constraints. Results show that software-level optimizations are the primary factors determining inference viability, often outweighing raw computational throughput. While modern GPUs benefit from optimized attention mechanisms and improved energy efficiency, legacy and lower-power devices remain viable when combined with optimized runtimes and model compression techniques. These findings demonstrate that contemporary generative workloads can be effectively supported by decentralized edge infrastructures, providing practical insights for the design of energy-efficient and heterogeneous local AI systems.
High-fidelity performance simulators are essential for designing and configuring efficient AI systems, yet today's tools lack the ability to predict power consumption. Established GPU power models rely on hardware utilization counters, which do not exist until the workload has actually run. This work evaluates whether Model FLOPs Utilization (MFU)-an analytical, software-defined metric relating achieved throughput to peak hardware capability-can serve as a portable, software-defined predictor of GPU power for LLMs. We benchmark almost 3000 single-device training runs across six GPUs, covering different model families, numerical precisions, batch sizes, and context-window lengths. We find that a linear MFU-based power model fits every tested GPU as long as the workload is compute-bound, as in production LLM training. Fitting per-(GPU, dtype, batch size) instead of per-GPU drops the within-cell mean error from around 10% to around 1%, matching the cross-repeat measurement-noise floor.
GPU energy consumption represents a primary operational cost for Large Language Model (LLM) inference services. Despite the inherent variability of production workloads, most deployments rely on static power configurations that fail to exploit the non-linear relationship between power limits and performance. Consequently, opportunities to reduce energy overhead while maintaining Service Level Agreements (SLAs) are often missed. While modern GPUs provide hardware-level powercapping interfaces, their systematic application in a feedbackdriven manner to satisfy stringent latency guarantees remains significantly underexplored. This paper proposes a lightweight, non-intrusive adaptive GPU power-capping mechanism based on a PID controller that dynamically adjusts power limits using SLA feedback. The controller operates directly through vendor power-limit interfaces and incorporates guardrails such as deadbands, rate limiting, utilization-aware gating, and antiwindup protection. We evaluate the approach on a server with eight NVIDIA H200 GPUs running the qwen3:32b model under a time-varying Poisson workload, with SLA compliance defined over TTFT. Results show that static caps expose a rigid energy-SLA trade-off, whereas the adaptive controller achieves a 3.84% reduction in total energy consumption relative to a 700 W baseline while maintaining empirical SLA non-inferiority. These findings demonstrate that feedback-driven GPU power modulation can improve energy efficiency without compromising latency stability.
DEFT is presented, an energy-aware scheduling framework that jointly optimizes task-to-device assignment and per-GPU DVFS configuration for task-based multi-GPU applications and employs a cost-model–driven strategy that integrates slack awareness, throughput awareness, and explicit modeling of task execution cost, inter-GPU data movement, and DVFS transition overheads, enabling coordinated placement and frequency decisions under dynamic runtime conditions.
With the rapid growth of artificial intelligence (AI) and high-performance computing (HPC), GPUs and other accelerators have become a shared computing substrate for a wide range of workloads. However, many shared accelerator clusters still rely on coarse device-level allocation, which often leads to low effective utilization, resource fragmentation, and long queueing delays. Although pooling technologies offer a promising direction, existing approaches remain limited in supporting fine-grained, low overhead sharing across heterogeneous accelerators and diverse co-located workloads. This paper presents gPooling, a hardware-agnostic accelerator pooling framework based on driver-level interception. gPooling creates elastic virtual devices on demand and extends fine-grained sharing across heterogeneous accelerators through a unified control path. We evaluate gPooling using benchmarks derived from real cluster traces and through deployment in a production GPU cluster. Results show that gPooling improves accelerator utilization, reduces user waiting time, and increases the overall efficiency of shared accelerator environments.
The deployment of real-time artificial intelligence inference services on shared cloud infrastructure poses significant challenges due to resource contention, latency variability, and tail-latency amplification. While cloud platforms offer scalability and flexibility, conventional accelerator sharing mechanisms often fail to provide the determinism required by latency-sensitive inference workloads. This paper presents a standards-based, multi-tenant cloud inference framework that integrates OpenStack orchestration with Single Root I/O Virtualization (SR-IOV)-enabled graphics processing unit (GPU) partitioning to achieve predictable and isolated real-time inference execution. In the proposed architecture, each tenant is assigned an exclusive GPU virtual function, enabling hardware-level isolation while remaining fully compatible with native OpenStack scheduling and resource management mechanisms. A comprehensive experimental evaluation is conducted on a private OpenStack cloud to assess inference latency distribution, tail behavior, scalability, robustness to background network and control-plane activity, and throughput-latency trade-offs. Experimental results show that median inference latency remains stable across single-tenant and multi-tenant configurations, while P95 and P99 tail latencies exhibit no measurable amplification under concurrent execution. The system scales linearly with the number of available GPU virtual functions, maintaining consistent latency behavior until hardware capacity is reached. Additional experiments demonstrate that background network traffic and control-plane operations introduce negligible impact on inference latency. Throughput analysis reveals a well-defined saturation knee, enabling clear identification of safe operating regions for real-time inference services. By leveraging mature ICT standards and open-source cloud infrastructure, this work provides a reusable reference architecture for deploying latency-sensitive inference services in private and hybrid clouds. The results highlight the effectiveness of hardware-assisted accelerator isolation in balancing performance determinism, scalability, and operational simplicity, and offer practical guidance for future system design and standardization efforts.