Synchronous Distributed Training With Runtime-Adaptive Mechanisms in Hybrid Cloud Environments

Abstract

The rapid growth of large datasets and AI models has necessitated scalable and powerful computing resources, driving the extension of training workloads to the cloud to balance cost and performance. Among distributed learning strategies, synchronous data-parallel training remains the most widely adopted approach due to its simplicity and stable convergence. However, heterogeneity between worker nodes and unreliability of network bandwidth pose substantial challenges. Since workers must synchronize gradients at every update step, even a single straggler can significantly degrade overall performance. In this paper, we present ASTRA, an auto-adaptive synchronous training framework tailored for hybrid cloud environments. ASTRA addresses these issues with three coordinated mechanisms: i) a network-topology detection module that clusters workers into groups by pairwise connection cost to enable hierarchical aggregation, where gradients are first aggregated within each group and then across groups, mitigating slow-link bottlenecks; ii) a load balancing module to efficiently distribute workloads among heterogeneous GPUs over time, reducing resource imbalance and straggler effects; and iii) an adaptive gradient compression module that dynamically adjusts compression levels based on network conditions to minimize inter-group communication overhead. Through automated adaptation to diverse system environments, our unified approach alleviates the burden of manual configuration for AI practitioners. Experimental evaluations demonstrate the effectiveness of the framework ASTRA, which achieves lower time-to-accuracy than a resource-heterogeneity-aware baseline and several compression-based frameworks, while preserving convergence quality and robustness across heterogeneous hybrid cloud environments.