Skip to content
Open access

OpDisPath: Optimal Communication Scheduling in GPU Clusters Through Disjoint Path Selection

2026 · IEEE Transactions on Networking · Vol 34, pp. 8011-8026 · 0 citations · 48 references

Abstract

The rapidly growing demand for distributed training has motivated providers to deploy large-scale GPU clusters. As a provider, we manage multiple clusters that hosts numerous distributed model training jobs that generate voluminous, highly dynamic concurrent flows. These flows may traverse the same link, causing severe bandwidth contention. Existing efforts attempt to minimize bandwidth contention by selecting least-congested links for each flow and prioritizing high-priority flows, yet they fail to eliminate it entirely. Our analysis shows that even a minimal number of flows under bandwidth contention may still cause severe straggler problems, significantly impairing communication time. To bridge this gap, we design OpDisPath, an approach that selects link-disjoint paths for the collective flows in each communication stage. Specifically, most GPU clusters are built on non-oversubscribed networks, i.e., the total uplink bandwidth of a switch is no less than the total downlink bandwidth. We prove that for such networks, there exists an optimal communication scheduling scheme capable of completely avoiding bandwidth contention among the scheduled flows in each collective-communication stage. To achieve this goal, we devise a disjoint path selection algorithm based on bipartite graph matching and provide theoretical proofs of its algorithmic optimality. Experimental validation on a physical testbed and through large-scale simulations of a 2,016-GPU cluster demonstrates that OpDisPath eliminates all bandwidth contention within each communication stage and achieves up to 77.5% reduction in iteration time compared to state-of-the-art methods.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.