AI training and inference are driving cloud networks toward terabit-per-second (Tbps) bandwidth per server, challenging the scalability and efficiency of today's cloud network architectures. A prevalent design scales bandwidth by stacking monolithic Data Processing Units (DPUs), but this approach tightly couples control and data plane resources, leading to excessive cost, power consumption, and operational complexity. We identify a fundamental control-data plane divergence in AI clouds: while data plane bandwidth demand grows rapidly, control plane demand remains largely flat due to the dominance of elephant flows. As a result, monolithic DPUs become systematically over-provisioned when used as bandwidth scaling primitives. We present DistDPU, a disaggregated DPU architecture that redefines the scaling abstraction for high-bandwidth cloud networking. DistDPU decomposes a monolithic DPU into lightweight, bandwidth-provisioning Execution Modules (EMs) and a shared, control-centric Orchestration Module (OM), enabling independent scaling of data and control plane resources. By scaling out low-cost EMs under a single OM, DistDPU exposes a unified, high-bandwidth logical DPU interface to the cloud management plane. To preserve RDMA performance and multi-tenant isolation at scale, we co-design the EM-OM functions and the inter-module fabric to minimize virtualization overhead while enforcing security and manageability invariants equivalent to those of a monolithic DPU. DistDPU has been deployed in production for two years. It serves more than 10,000 GPUs and delivers higher efficiency and strong performance on real-world AI workloads than state-of-the-art designs.
Hao Mei, Lizhou Gao, Yuanyi Zhu et al.· 0 citations
Modern cloud gateways have evolved to include diverse network functions and heterogeneous hardware, such as programmable switches and FPGAs, to handle increasing workloads and minimize forwarding latency. Existing network tracing tools, however, operate primarily at device granularity and cannot pinpoint which function on which hardware component causes packet losses or latency spikes. To bridge this gap, we present CubeTrace, a unified, function-level flow tracing system that enables microscopic tracing inside heterogeneous cloud gateways. CubeTrace standardizes tracing units as cubes across different hardware platforms, regardless of their varied underlying implementations, and operates at flow granularity for reliability reasons. This introduces a new tracing abstraction for heterogeneous gateways while maintaining low overhead. Moreover, the collected flow-cube data by CubeTrace can be decoded into packet-level representations and integrated with well-established distributed tracing frameworks, enabling the use of off-the-shelf analysis tools. Our evaluations demonstrate that CubeTrace introduces minimal overhead, consuming less than 1% of memory resources and adding less than 1% to forwarding latency. Having been deployed in a large-scale cloud gateway, CubeTrace has significantly improved problem localization, reducing resolution times from hours or even days to just minutes.
Yunming Xiao, Yinchao Yang, Jiaqi Zheng et al.· Proceedings of the ACM SIGCO...· 1 citation
The experience in designing, deploying, and operating Pegasus, a data center network tailored for the AI cloud, along with operational lessons learned from its deployment are shared.
Xianneng Zou, Yadong Liu, Yiran Zhang et al.· Proceedings of the ACM SIGCO...· 0 citations