Fumer: Proactive Time-Shifting for Synchronized Periodic Traffic in Distributed Training

Abstract

The growth of distributed training models, with parameters now reaching the billion-scale, has shifted the system bottleneck from computation to communication. While Remote Direct Memory Access (RDMA) is widely deployed to improve network performance by circumventing the kernel mechanism, the synchronization-computation cycles under the synchronous parallel mode introduce a highly synchronized and periodic “on-off” bursty traffic pattern, which poses significant challenges to data center networking. Consequently, distributed training suffers from two critical bottlenecks: instantaneous congestion during communication and persistent link idleness during computation. These issues lead to severe bandwidth contention and resource underutilization, ultimately hindering overall training efficiency. To address these challenges, this paper proposes Fumer, a proactive periodic traffic optimization framework that shifts the congestion control paradigm from reactive rate adjustment to proactive time-shifting. Specifically, Fumer leverages In-band Network Telemetry (INT) and Fast Fourier Transform (FFT) with signal-wave separation to decompose interleaved traffic signals, aiming to overcome the lack of periodic awareness. Furthermore, Fumer employs an off-peak transmission optimization algorithm to calculate optimal time-shift values, thereby tackling synchronized congestion and link idleness. By executing proactive off-peak scheduling, Fumer shifts overlapping communication windows into idle periods to smooth traffic peaks in the time domain. Experimental results show that Fumer boosts average path throughput across all workloads to 86.3 Gbps, improving upon DCQCN (42.6 Gbps) by 102.6% and RECC by 22.1%. Furthermore, it reduces the average and 99.9th-percentile iteration times by up to 25.0%-45.4% and 25.5%-49.3%, respectively, demonstrating its efficacy and robustness across diverse large-scale training workloads.