Ghost is an OS-level GPU virtualization layer integrated directly into the open-source GPU driver, using a GPU container abstraction with cgroup -like APIs for compute and memory control and privileged hardware-level scheduling and preemption for dynamic compute resource management.
With the rapid growth of artificial intelligence (AI) and high-performance computing (HPC), GPUs and other accelerators have become a shared computing substrate for a wide range of workloads. However, many shared accelerator clusters still rely on coarse device-level allocation, which often leads to low effective utilization, resource fragmentation, and long queueing delays. Although pooling technologies offer a promising direction, existing approaches remain limited in supporting fine-grained, low overhead sharing across heterogeneous accelerators and diverse co-located workloads. This paper presents gPooling, a hardware-agnostic accelerator pooling framework based on driver-level interception. gPooling creates elastic virtual devices on demand and extends fine-grained sharing across heterogeneous accelerators through a unified control path. We evaluate gPooling using benchmarks derived from real cluster traces and through deployment in a production GPU cluster. Results show that gPooling improves accelerator utilization, reduces user waiting time, and increases the overall efficiency of shared accelerator environments.
Kaicheng Guo, Jingyi Chen, Chen Chen et al.· IEEE Transactions on Paralle...· 0 citations
This work proposes SAI, a mechanism that virtualizes shared memory into the L2 cache to improve GPU performance for AI applications and introduces an L2 cache management strategy that integrates associativity-based virtual page allocation and a replacement information table, reducing page-swapping overhead while preserving L2 cache performance.
Hanqing Li, Tiejun Li, Sheng Ma et al.· ACM Transactions on Design A...· 0 citations
High-performance GPU programming has traditionally forced a compromise between execution efficiency and memory safety. While Rust guarantees compile-time memory safety for host CPUs via its strict ownership model, applying these constraints to massively parallel GPU execution environments has previously mandated either vendor-locked Domain-Specific Languages (DSLs) or escaping to explicit unsafe raw pointers. This paper presents a zero-overhead, multi-vendor GPU compilation framework built natively into the Rust compiler (rustc) and LLVM backends. We leverage Rust's rich type system, ownership system, and strict aliasing guarantees (noalias) to efficiently manage and optimize data transfers through LLVM's Offload infrastructure. We expose the technical challenges of cross-vendor ABI lowering mismatches between Host and Device targets and introduce a two-pass compilation pipeline capable of safely handling both manual and compiler-generated memory movements. Evaluating our framework on RAJAPerf demonstrates that our rustc-based solution can generate competitive LLVM IR for GPU kernels, achieving a solid kernel performance against native, hand-optimized CUDA and HIP C++ baselines.
Manuel S. Drehwald, Marcelo Domínguez, Kevin Sala et al.· 0 citations
ElastiCo is presented, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms that decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices.
Jinghao Wang, Yihang Zhou, Xiaoyang Sun et al.· 0 citations
Thanks to notable performance and capacity advantages, NVMe SSDs promise an effective cache tier for alleviating the load pressure of back-end storage servers. Compared with DRAM, an NVMe SSD exhibits up to hundreds of times larger capacity but delivers two orders of magnitude smaller bandwidth per gigabyte. This paper reveals that these hardware characteristics challenge fundamental design goals of caches: (1) storing tons of objects in a large-capacity SSD easily induces severe, persistent fragmentation, resulting in low cache space utilization; (2) intensive front-end requests contend for limited SSD bandwidth, causing unpredictable cache lookup latency. To tackle these challenges, this paper presents Gemini. The core of Gemini is heat-informed, group-based object management with a hardware-assisted I/O prioritization mechanism. Specifically, we introduce a tree-based prediction model for agile object grouping and fast reaction to hotspot shift. It provides SSD-friendly, bulk object eviction with a zero-write, remapping-based migration mechanism. In addition, to ensure request performance, we propose a novel I/O model that mitigates bandwidth congestion and prioritizes their processing at the hardware layer. Evaluation results show that Gemini delivers up to 3.8<inline-formula><tex-math notation="LaTeX">${\boldsymbol{\times}}$</tex-math><alternatives><mml:math><mml:mrow><mml:mo mathvariant="bold">×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="cai-ieq1-3711088.gif"/></alternatives></inline-formula> throughput improvement, 2<inline-formula><tex-math notation="LaTeX">${\boldsymbol{\times}}$</tex-math><alternatives><mml:math><mml:mrow><mml:mo mathvariant="bold">×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="cai-ieq2-3711088.gif"/></alternatives></inline-formula> tail latency reduction, and 2.7<inline-formula><tex-math notation="LaTeX">${\boldsymbol{\times}}$</tex-math><alternatives><mml:math><mml:mrow><mml:mo mathvariant="bold">×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="cai-ieq3-3711088.gif"/></alternatives></inline-formula> higher cache space utilization than state-of-the-art caches for both YCSB workloads and production applications.
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
Shuo Yang, Xiao-yun Fan, Melissa Z. Pan et al.· 2 citations