The performance of datacenter congestion control algorithms (CCAs) is highly sensitive to bursty traffic patterns, yet a significant fidelity gap exists between evaluation workloads and production traffic. Current evaluations primarily rely on synthetic workloads constructed from flow-size CDFs with incast overlaid on top, an approach that, while intuitive, we show produces traffic that is dissimilar to production in its temporal burst clustering. As a result, these workloads fail to exercise the full range of conditions that CCAs encounter in production, and hence, protocols that demonstrate gains in simulation risk diminished performance or unexpected failure modes upon deployment. This motivates the need for a deeper understanding of burstiness for CCA evaluation. To this end, we decompose burstiness into four key dimensions, and use DCTCP as a case study to show distinct behavioral regimes in each dimension. Building on this, we envision a burst-centric evaluation stack: behavioral regime analysis across various CCA classes, and a burst generator to ensure regime coverage along these dimensions, enabling thorough and robust evaluations.
Pragna Mamidipaka, Srikanth Sundaresan, Theophilus A. Benson· Asia-Pacific Workshop on Net...· 0 citations
The arrival of 100K+ GPU clusters marks a new frontier in AI infrastructure. Standard communication stack meets new challenges as physical topologies span multiple datacenter buildings, introducing high bandwidth-delay product links where latency increases by up to 30× compared to intra-rack traffic. Furthermore, the transition toward Mixture-of-Experts architectures generating bursty all-to-all patterns that create transient congestion hotspots. These constraints, combined with an operational environment where hardware failures shift from anomalies to frequent occurrences, renders traditionally lightweight operations like initialization and resource management challenging. We present Meta's network architecture and software stack designed to support one of the world's largest RoCE fabrics, currently connecting over 100,000 GPUs across multiple datacenter buildings. To overcome scaling barriers, we introduce a scalable initialization strategy that reduces startup times by 11× via eager process group creation and O(N) topology discovery, alongside a resource management system that cuts GPU memory usage by 2× through on-demand allocation. We further detail a custom transport layer utilizing Dynamic Queue Pair Load Balancing to saturate links, and a set of operation toolings. These innovations have been deployed in production, providing the foundational communication fabric for training state-of-the-art Large Language Models.
Hongyi Zeng, Min Si, Pavan Balaji et al.· Proceedings of the ACM SIGCO...· 0 citations
