Gleam is a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions that reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution.
Abstract
This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneous personal devices. We achieve distributed task offloading via CUDA API remoting. However, beyond raw computation, network constraints emerge as the primary bottleneck: limited bandwidth, high-frequency API invocations, and cross-task contention significantly hinder performance. To address these challenges, we propose Gleam, a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions. First, we reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution. Second, we design a runtime task scheduler that dynamically determines API remoting pairs between LAN clients and servers, explicitly accounting for both network conditions and GPU resource contention under parallel workloads. Finally, we introduce dedicated mechanisms to ensure CUDA context consistency across distributed executions. Extensive experiments on heterogeneous NVIDIA GPUs and diverse AI workloads show Gleam consistently outperforms state-of-the-art baselines, achieving 1.4-24.2 times improvements in API remoting efficiency and up to 1.79 times higher system throughput.
ElastiCo is presented, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms that decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices.
Jinghao Wang, Yihang Zhou, Xiaoyang Sun et al.· 0 citations
Graphics processing unit (GPU) architectures are growing in size to meet the increasing compute and memory requirements. As GPU sizes increase, intra-socket wire transfer delay increases significantly. While previous research has optimized for compute and memory locality within a socket, the spatial impact on inter-GPU communication has not been well-studied. We introduce the term non-uniform network access (NUNA) to describe this emerging optimization dimension in multi-GPU systems. We specifically focus on latency-sensitive collective communication, common in machine learning inference. First, we highlight the need for NUNA-aware routing (NAR), choosing optimized, spatially-aware inter-GPU paths in large scale-up network topologies. Second, we introduce NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic. We demonstrate that the NAP optimizations alone offer up to 1.5x collective speedups over a locality-unaware baseline. Combining NAP with NAR yields up to 1.8x faster collectives over the locality-unaware baseline. This leads to 7% mean (28% max) time per output token speedup in machine learning inference.
With the rapid growth of artificial intelligence (AI) and high-performance computing (HPC), GPUs and other accelerators have become a shared computing substrate for a wide range of workloads. However, many shared accelerator clusters still rely on coarse device-level allocation, which often leads to low effective utilization, resource fragmentation, and long queueing delays. Although pooling technologies offer a promising direction, existing approaches remain limited in supporting fine-grained, low overhead sharing across heterogeneous accelerators and diverse co-located workloads. This paper presents gPooling, a hardware-agnostic accelerator pooling framework based on driver-level interception. gPooling creates elastic virtual devices on demand and extends fine-grained sharing across heterogeneous accelerators through a unified control path. We evaluate gPooling using benchmarks derived from real cluster traces and through deployment in a production GPU cluster. Results show that gPooling improves accelerator utilization, reduces user waiting time, and increases the overall efficiency of shared accelerator environments.
Kaicheng Guo, Jingyi Chen, Chen Chen et al.· IEEE Transactions on Paralle...· 0 citations
HyLink is introduced, a two-layer collective communication framework, including a PCIe CCL module and a Multipath Orchestrator module that provides a CPU-driven Domain-Specific Language (DSL) for multi-stage data flows and an auto-pipelined transfer engine for host-side routing and SIMD-accelerated reduction.
Yue Liu, Menghao Zhang, Xuebin Song et al.· Asia-Pacific Workshop on Net...· 0 citations
Ghost is an OS-level GPU virtualization layer integrated directly into the open-source GPU driver, using a GPU container abstraction with cgroup -like APIs for compute and memory control and privileged hardware-level scheduling and preemption for dynamic compute resource management.
The rapid growth of large datasets and AI models has necessitated scalable and powerful computing resources, driving the extension of training workloads to the cloud to balance cost and performance. Among distributed learning strategies, synchronous data-parallel training remains the most widely adopted approach due to its simplicity and stable convergence. However, heterogeneity between worker nodes and unreliability of network bandwidth pose substantial challenges. Since workers must synchronize gradients at every update step, even a single straggler can significantly degrade overall performance. In this paper, we present ASTRA, an auto-adaptive synchronous training framework tailored for hybrid cloud environments. ASTRA addresses these issues with three coordinated mechanisms: i) a network-topology detection module that clusters workers into groups by pairwise connection cost to enable hierarchical aggregation, where gradients are first aggregated within each group and then across groups, mitigating slow-link bottlenecks; ii) a load balancing module to efficiently distribute workloads among heterogeneous GPUs over time, reducing resource imbalance and straggler effects; and iii) an adaptive gradient compression module that dynamically adjusts compression levels based on network conditions to minimize inter-group communication overhead. Through automated adaptation to diverse system environments, our unified approach alleviates the burden of manual configuration for AI practitioners. Experimental evaluations demonstrate the effectiveness of the framework ASTRA, which achieves lower time-to-accuracy than a resource-heterogeneity-aware baseline and several compression-based frameworks, while preserving convergence quality and robustness across heterogeneous hybrid cloud environments.
Tuan Anh Vuong, Thanh Loi Hoang, Huan Le et al.· IEEE Access· 0 citations