Jun 2026· International Symposium on Computer Architecture· pp. 2142-2157· 0 citations· 67 references
Abstract
Efficient collective communication is crucial for distributed training. While recent topology-aware synthesis approaches attempt to optimize communication based on the network topology, they struggle with heterogeneous environments where links exhibit distinct bandwidths and asymmetric connections. The main inefficiency arises from single-round communication: toward the end of a communication phase, only a few remaining transfers occupy limited links, leaving most network channels idle. Furthermore, existing methods fail to handle the congestion that naturally occurs under overlapping execution, leading to significant bandwidth underutilization. To address these challenges, we propose PipeComm, a pipelineaware communication synthesis framework that maximizes bandwidth utilization by intelligently overlapping multiple data chunks. By explicitly modeling pipeline behavior, our method enables congestion-free scheduling across iterations and efficiently utilizes heterogeneous links. We develop an optimal synthesis method for constructing high-quality communication patterns and introduce a complementary incremental strategy that significantly improves scalability for large topologies. Experimental results show that the optimal strategy achieves over a 1.39× speedup compared to the state-of-the-art communication methods. Moreover, PipeComm supports diverse collective operations, demonstrating both efficiency and generality.
ReTri, a bidirectional All-to-All schedule for ORNs based on the Trivance algorithm is presented, a bidirectional All-to-All schedule for ORNs based on the Trivance algorithm that improves completion time and improves reconfigurable Bruck by up to 2.1×.
Anton Juerss, Stefan Schmid· Proceedings of the ACM SIGCO...· 0 citations
Co is proposed, a coherence-aware co-optimization framework that jointly integrates task mapping and routing under a unified cost model for realistic scenarios, enabling coherence-aware decision-making and effective trade-offs among optimization goals.
HyLink is introduced, a two-layer collective communication framework, including a PCIe CCL module and a Multipath Orchestrator module that provides a CPU-driven Domain-Specific Language (DSL) for multi-stage data flows and an auto-pipelined transfer engine for host-side routing and SIMD-accelerated reduction.
Yue Liu, Menghao Zhang, Xuebin Song et al.· Asia-Pacific Workshop on Net...· 0 citations
Decentralized large language model (LLM) inference distributes transformer layers across heterogeneous nodes to enable resource-constrained participants to collaboratively serve large models. However, inter-node activation transfer constitutes a significant communication bottleneck, especially under low-bandwidth conditions prevalent in consumer-grade and edge networks. We propose BandwidthLLM, a communication-efficient framework that integrates three techniques: (1) a bandwidth-aware layer placement algorithm that minimizes boundary-level transfer cost according to link bandwidth and node reliability; (2) a lightweight activation compression scheme combining adaptive quantization with outlier-aware clipping and error feedback; and (3) a semantic preservation check that automatically falls back to higher precision when compressed activations deviate beyond a calibrated threshold. Experiments on Llama-2-7B, Mistral-7B, and Qwen2.5-7B across simulated bandwidth conditions (20–500 Mbps) demonstrate that BandwidthLLM reduces inter-node activation traffic by 62.7–63.4% relative to the BloomBee-like baseline while maintaining perplexity degradation of at most 0.31 points and task accuracy within 0.9 percentage points of uncompressed inference. Under 20 Mbps links, BandwidthLLM achieves 1.7× higher throughput and 41% lower latency.
Zhihao Wang, Shi Shu, Yulin Wang et al.· 2026 8th International Confe...· 1 citation
The Message Passing Interface (MPI) is a de-facto standard programming model for developing high-performance applications on large-scale supercomputers, where the performance of its collective communication operations over the underlying interconnection network is critical to the overall scalability of the applications. While the MPI defines the semantics of these operations, it leaves the algorithmic implementation to Open MPI, and selecting the algorithm typically relies on hard-coded decision logic. However, in real-world scenarios, this static approach often leads to sub-optimal performance, as it ignores the specific properties of the underlying system, particularly the complex and dynamic interconnection topologies. In this work, we address the challenge of selecting an efficient algorithm for a given collective, taking into account interconnection topologies. First, we demonstrate through a simulation study that interconnect topological features significantly influence the choice of the optimal collective algorithm for specific communication instances. Second, we propose a novel, machine learning-based, topology-aware framework for MPI collective algorithm selection. Third, we integrate this predictive framework directly into Open MPI, and show that our method is effective in selecting efficient collective algorithms for the Next Generation Tianhe Supercomputer, thereby unlocking the performance potential of the underlying interconnect.
Meng-Cheng Jia, Wei Zhang, Hui-Ze Zhang et al.· Fall Joint Computer Conferen...· 0 citations
This paper presents a tutorial-style, collective-centric taxonomy for collective communication, which organizes recent advances into three layers: communication planning, which generates topology-aware collective schedules; communication execution and adaptation, which maps these schedules onto GPU runtimes and hardware in real clusters; and computation-communication coordination, which turns collective optimization into end-to-end training and inference benefits.
Xuebin Song, Menghao Zhang, Yue Liu et al.· IEEE Network· 0 citations