Aug 2026· Asia-Pacific Workshop on Networking· 0 citations· 36 references
Computer Science
TL;DR
HyLink is introduced, a two-layer collective communication framework, including a PCIe CCL module and a Multipath Orchestrator module that provides a CPU-driven Domain-Specific Language (DSL) for multi-stage data flows and an auto-pipelined transfer engine for host-side routing and SIMD-accelerated reduction.
Abstract
Distributed LLM training requires GPUs to cooperate through collective communication primitives. While systems with proprietary high-speed interconnects have been extensively studied, the PCIe bandwidth in GPU clusters remains severely underutilized. Existing collective communication libraries treat the hierarchical PCIe tree as a flat GPU-to-GPU network and relegate the CPU to a passive data relay. Moreover, in systems equipped with dedicated interconnects, the PCIe path is largely overlooked. This paper introduces HyLink, a two-layer collective communication framework, including a PCIe CCL module and a Multipath Orchestrator module. PCIe CCL unlocks underutilized PCIe bandwidth by natively modeling the PCIe hierarchy and elevating the CPU to an active routing and computation node. It provides a CPU-driven Domain-Specific Language (DSL) for multi-stage data flows and an auto-pipelined transfer engine for host-side routing and SIMD-accelerated reduction. Multipath Orchestrator further enables hybrid-link acceleration by splitting each collective operation across the dedicated link and the PCIe bus. It employs a feedback-driven adaptive balancer to dynamically adjust the load ratio to accommodate runtime PCIe interference. Our experiments show that PCIe CCL achieves up to 2.28 × the bandwidth of NCCL on PCIe-only NVIDIA GPUs. HyLink improves bandwidth by up to 40% over HCCL on Ascend NPUs, and Multipath Orchestrator achieves 66% higher throughput than static splitting under dynamic PCIe interference.
Modern collective communication libraries (CCLs) execute a collective communication task (CCT) by decomposing it into multiple sub-tasks, each mapped to a specific Virtual Topology (VT), which is an ordered graph of GPUs (e.g., a ring or a tree), to maximize parallelism and link utilization. As AI training scales to larger clusters, network anomalies (congestion and failures) are unavoidable, and a single straggling VT can delay the entire CCT. Existing solutions either rely on low-level transport-layer solutions which lacks a cross-sub-task perspective, or static CCL scheduling, failing to adapt to the dynamic and heterogeneous networks. We present PReCCL, a drop-in NCCL replacement that combines software inband telemetry with cross-VT workload reallocation. Unlike hardware-dependent telemetry (e.g., P4-INT), PReCCL implements in-band monitoring within the CCL. It precisely measures the stall counts of each VT, and piggybacks the telemetry meta-data on existing collective traffic. With this fine-grained visibility, PReCCL employs a dynamic cross-VT reallocation mechanism that shifts workloads from straggler VTs (suffering from congestion or faults) to healthy ones. Reallocation occurs only at CCT boundaries using an epoch-based deterministic protocol, avoiding cross-rank consistency races. We implement PReCCL in NCCL and evaluate it on a 32-GPU testbed, large-scale simulation, and a 1024-GPU production cluster. On the testbed, PReCCL improves CCT completion by up to 2.1X and end-to-end training by 1.21X, with low overhead above tuned message-size thresholds.
Zhiyong Chen, Kaihui Gao, Li Chen et al.· Proceedings of the ACM SIGCO...· 0 citations
Gleam is a novel and network-efficient framework for task-generic GPU sharing across local-area CUDA devices, with three key contributions that reduce bandwidth overhead in CUDA API remoting through automatic model weight caching, and mitigate accumulated latency from frequent API calls by asynchronous execution.
Zhihao Xu, Hao Zhong, Zeting Zhou et al.· 0 citations
ReTri, a bidirectional All-to-All schedule for ORNs based on the Trivance algorithm is presented, a bidirectional All-to-All schedule for ORNs based on the Trivance algorithm that improves completion time and improves reconfigurable Bruck by up to 2.1×.
Anton Juerss, Stefan Schmid· Proceedings of the ACM SIGCO...· 0 citations
GPU memory offloading is widely adopted for LLM workloads but shifts the bottleneck to GPU-CPU transfers, which can take up to 90% of the end-to-end inference/training time! Paradoxically, over 60% of PCIe bandwidth remains idle. The root cause is that PCIe links are individually bottlenecked but collectively underutilized. Bursty, phase-driven transfer patterns leave bandwidth idle both within and across jobs. We present TurboBus, which pools PCIe bandwidth across co-located jobs via emerging scale-up fabrics. TurboBus enables any GPU to borrow idle PCIe links from neighboring GPUs, even those belonging to other jobs, while preserving isolation through a privileged daemon. At the core of TurboBus, it streams data through relay GPUs with bounded memory overhead, keeps all links busy through fine-grained PCIe allocation, enables bidirectional transfers leveraging PCIe/NVLink bandwidth asymmetry, and balances fairness and completion time with a size-aware scheduling. We fully implement TurboBus and our experiments show that it reduces first-token latency by up to 40% for on-demand model loading (within 5% of the analytical optimum), achieves up to 1.6x throughput for KV-cache-offloaded inference, and accelerates training by up to 7%, while imposing less than 1% overhead on co-located workloads.
Xinyu Yang, Kaiqiang Xu, Kai Chen· Proceedings of the ACM SIGCO...· 0 citations
We present HCCL, a collective communication library co-designed with Meta's MTIA 300 accelerator, the first Meta chip to integrate backend networking directly on chip package. MTIA 300 includes dedicated message engines (MEs) with near-memory compute (NMC) that fully offload collective execution from the compute grid, enabling large overlap between computation and communication. HCCL uses a compiled communication model in which the host generates a complete description of each collective including dependencies. We describe the control and data path architecture, topology-aware algorithm selection across MTIA 300's asymmetric scale-up and scale-out network, and optimizations for both training and inference workloads. For training, HCCL achieves up to 940 GB/s on intra-rack collectives while introducing less than 0.5% degradation to concurrent compute throughput. For inference, we leverage one-sided communication primitives that bypass the scheduling path to minimize collective latency and describe collective designs that improve compute-communication pipelining for latency-sensitive workloads.
W. Bland, Tiago Antunes, Lars Paul Huse et al.· 0 citations
Rack-scale interconnects serve as critical datapaths for emerging communication-intensive systems to scale up. Innovative solutions for this datapath are rising at a rapid pace, especially those based on Ethernet. However, existing hardware-based solutions, such as RDMA, face performance issues, particularly for small-message memory access, and suffer from the inflexibility of hardware-fixed processing. The community is actively pursuing efficient, flexible, and cost-effective rack-scale datapaths. In this work, we propose Software-Interposed Datapath (SID), an efficient, software-flexible, and low-cost solution for rack-scale interconnects, particularly optimized for fine-grained memory access. Improving small-message efficiency is a well-known challenge, and software involvement for flexibility seems to amplify the performance hurdle further. SID boosts performance by exploiting one insight: existing NICs primarily rely on Queue Pair (QP)-level parallelism, but underutilize intra-QP Work Queue Element (WQE)-level parallelism. Harnessing parallelism is non-trivial, especially at the WQE-level, due to ordering semantics and request dispatching. The key technique is our Elastic QP data structure built atop the on-NIC datapath processors, which realizes ordered intra-QP parallelism while minimizing coordination overhead. Regarding flexibility, SID supports extensible operation sets that comply with the OpenSHMEM model for ML/HPC workloads and the Message Queue model for cloud service workloads. Regarding cost efficiency, SID is built on top of commodity components such as Ethernet, PCIe, and datapath cores of NVIDIA ConnectX-8 and BlueField-3 NICs. Evaluation shows that SID achieves up to 11.03x higher rates for small messages than RDMA-based baselines and supports both CPU and GPU-Direct operations.
Chenxingyu Zhao, Yibo Wu, Hongtao Zhang et al.· Proceedings of the ACM SIGCO...· 0 citations