A structured, paired comparison of configuration complexity and performance across six platforms (Proxmox VM and LXC, native KVM, OpenStack VM and Zun, and Podman) on a single NVIDIA RTX 4500 Ada GPU, emphasizing the under-documented container paths
Abstract
Enterprises increasingly run GPU-bound workloads such as Large Language Model (LLM) inference inside virtualized infrastructure, yet practitioners have little systematic guidance on how the choice of virtualization platform affects GPU passthrough in configuration effort as much as performance. This study’s primary contribution is a structured, paired comparison of configuration complexity and performance across six platforms (Proxmox VM and LXC, native KVM, OpenStack VM and Zun, and Podman) on a single NVIDIA RTX 4500 Ada GPU, emphasizing the under-documented container paths (OpenStack Zun and Podman with the Container Device Interface). The platforms differ most in configuration complexity, where OpenStack, especially Zun, proved the most demanding and Proxmox and LXC the most straightforward. As a confirmatory baseline, for a small-model, low-concurrency vLLM workload that does not saturate the device, exclusive passthrough yields statistically equivalent throughput, duration, and inter-token latency across all platforms, within 2% of bare metal; this equivalence is bounded to the non-saturating regime tested and is not a general claim. Running two containers concurrently (each capped at 45% of GPU memory, compute time-sliced) doubled the offered load and raised aggregate throughput by 12–14%, consistent with idle GPU capacity but not isolating a GPU-sharing benefit.
Heterogeneous DRAM-based processing-in-memory (PIM)-GPU systems promise significant efficiency gains for decode-phase large language model (LLM) inference, particularly in long-output generation, yet current design practices overlook critical factors that determine real-world performance. Through systematic evaluation of diverse architectures and workloads (OPT-7B/70B, Mamba2-2.7B/70B), we reveal three fundamental design principles: (i) static power consumption (DRAM leakage, refresh, and GPU idle power) can dominate the efficiency calculus, causing dynamic-only models to overestimate tokens/s/W by up to 3.85X for realistic deployments (Mamba2-2.7B, batch size 1, 128 input tokens, and 2,048 output tokens); (ii) decoding performance is monotonically non-decreasing with channel count across all evaluated models and workloads, generally plateauing at high channel counts for low-batch workloads; under a fixed-capacity sweep, all models instead share a common near-optimal hierarchy configuration, with substantially larger misconfiguration penalties for attention-based models; (iii) workload mapping strategies provide bounded improvements (up to 14.0%/17.4% kernel-level latency/energy reduction, up to 5.6% end-to-end gain) and are not primary bottlenecks. Significant efficiency gains require system-wide co-optimization. These principles provide design-space guidance for architects designing the next generation of memory-accelerated LLM systems.
Corey Lammie, Hadjer Benmeziane, W. Simon et al.· 0 citations
The deployment of real-time artificial intelligence inference services on shared cloud infrastructure poses significant challenges due to resource contention, latency variability, and tail-latency amplification. While cloud platforms offer scalability and flexibility, conventional accelerator sharing mechanisms often fail to provide the determinism required by latency-sensitive inference workloads. This paper presents a standards-based, multi-tenant cloud inference framework that integrates OpenStack orchestration with Single Root I/O Virtualization (SR-IOV)-enabled graphics processing unit (GPU) partitioning to achieve predictable and isolated real-time inference execution. In the proposed architecture, each tenant is assigned an exclusive GPU virtual function, enabling hardware-level isolation while remaining fully compatible with native OpenStack scheduling and resource management mechanisms. A comprehensive experimental evaluation is conducted on a private OpenStack cloud to assess inference latency distribution, tail behavior, scalability, robustness to background network and control-plane activity, and throughput-latency trade-offs. Experimental results show that median inference latency remains stable across single-tenant and multi-tenant configurations, while P95 and P99 tail latencies exhibit no measurable amplification under concurrent execution. The system scales linearly with the number of available GPU virtual functions, maintaining consistent latency behavior until hardware capacity is reached. Additional experiments demonstrate that background network traffic and control-plane operations introduce negligible impact on inference latency. Throughput analysis reveals a well-defined saturation knee, enabling clear identification of safe operating regions for real-time inference services. By leveraging mature ICT standards and open-source cloud infrastructure, this work provides a reusable reference architecture for deploying latency-sensitive inference services in private and hybrid clouds. The results highlight the effectiveness of hardware-assisted accelerator isolation in balancing performance determinism, scalability, and operational simplicity, and offer practical guidance for future system design and standardization efforts.
Mauricio Rui, Bingfeng Shi, Ma Xin et al.· Journal of ICT Standardizati...· 0 citations
High-performance GPU programming has traditionally forced a compromise between execution efficiency and memory safety. While Rust guarantees compile-time memory safety for host CPUs via its strict ownership model, applying these constraints to massively parallel GPU execution environments has previously mandated either vendor-locked Domain-Specific Languages (DSLs) or escaping to explicit unsafe raw pointers. This paper presents a zero-overhead, multi-vendor GPU compilation framework built natively into the Rust compiler (rustc) and LLVM backends. We leverage Rust's rich type system, ownership system, and strict aliasing guarantees (noalias) to efficiently manage and optimize data transfers through LLVM's Offload infrastructure. We expose the technical challenges of cross-vendor ABI lowering mismatches between Host and Device targets and introduce a two-pass compilation pipeline capable of safely handling both manual and compiler-generated memory movements. Evaluating our framework on RAJAPerf demonstrates that our rustc-based solution can generate competitive LLVM IR for GPU kernels, achieving a solid kernel performance against native, hand-optimized CUDA and HIP C++ baselines.
Manuel S. Drehwald, Marcelo Domínguez, Kevin Sala et al.· 0 citations
MaverIQ is an intent-based LLM inference serving system that automatically maps user intents to deployment configurations while minimizing operational cost for the provider and reduces profiling overheads by 7-15× compared to state-of-the-art baselines.
Dimitrios Liakopoulos, Prasoon Sinha, Tianrui Hu et al.· ACM SIGOPS Operating Systems...· 0 citations
OpScale is presented, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving that attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.
Xingqi Cui, Chieh-Jan Mike Liang, Ziang Tang et al.· 0 citations
KV-cache offload is widely used to stretch GPU memory for LLM serving, but its storage behavior has not been characterized at the block-device level. In this paper, we study LMCache through realworld multi-session workloads that span same/different context $\times$ same/different prompt, using over 100 stateless requests per workload. Our comparison is observational rather than factorial: it contrasts two realizable deployment snapshots—LMCache 0.3.0 with buffered I/O and the kernel page cache, and LMCache 0.3.16 with prefix-aware deduplication and O_DIRECT. Key findings define the paper. First, the legacy stack appears almost read-free at the SSD layer in all four workloads, but a flush_ram experiment shows that this is conditional on page-cache warmth: once the cache is evicted, the same path issues about 1 GB of sequential reads in 683 ms. Second, deduplication delivers a dramatic write reduction only for the exact-repeat workload: one set collapses from 1.62 GB to 11.75 MB of writes, while the other three sets remain in the 2–5 GB range. Third, O_DIRECT converts a hidden and bimodal read cost into an explicit and stable one: on one set, warm TTFT rises from 27 ms to 93 ms because each of the 99 warm requests re-reads the same ~12 MB partial chunk from SSD. The main conclusion is therefore not “newer is better,” but a system trade-off: predictability versus average-case latency.
Ying He, Dingsen Shi, Yanbo Dai et al.· 2026 International Conferenc...· 0 citations