AI training and inference are driving cloud networks toward terabit-per-second (Tbps) bandwidth per server, challenging the scalability and efficiency of today's cloud network architectures. A prevalent design scales bandwidth by stacking monolithic Data Processing Units (DPUs), but this approach tightly couples control and data plane resources, leading to excessive cost, power consumption, and operational complexity. We identify a fundamental control-data plane divergence in AI clouds: while data plane bandwidth demand grows rapidly, control plane demand remains largely flat due to the dominance of elephant flows. As a result, monolithic DPUs become systematically over-provisioned when used as bandwidth scaling primitives. We present DistDPU, a disaggregated DPU architecture that redefines the scaling abstraction for high-bandwidth cloud networking. DistDPU decomposes a monolithic DPU into lightweight, bandwidth-provisioning Execution Modules (EMs) and a shared, control-centric Orchestration Module (OM), enabling independent scaling of data and control plane resources. By scaling out low-cost EMs under a single OM, DistDPU exposes a unified, high-bandwidth logical DPU interface to the cloud management plane. To preserve RDMA performance and multi-tenant isolation at scale, we co-design the EM-OM functions and the inter-module fabric to minimize virtualization overhead while enforcing security and manageability invariants equivalent to those of a monolithic DPU. DistDPU has been deployed in production for two years. It serves more than 10,000 GPUs and delivers higher efficiency and strong performance on real-world AI workloads than state-of-the-art designs.
Hao Mei, Lizhou Gao, Yuanyi Zhu et al.· 0 citations
FPGA-based SmartNICs are widely deployed for cloud network function acceleration, but their memory subsystem is under increasing pressure because of large session tables. Conventional wisdom suggests that high packet processing performance relies on advanced memories (e.g., SRAM, HBM), but those are costly to add at cloud scale. Dorado is a novel design that scales SmartNIC session tables entirely on inexpensive DDR modules. At the heart of Dorado are three new techniques that extract commodity DDR performance by restructuring session table layout, decomposing processing pipelines to reduce locking, and scheduling memory accesses to minimize stalls. Our testbed results show that Dorado improves packet processing rates by 33%, even with fewer hardware resources. Further, we have deployed Dorado to millions of servers, processing network traffic from billions of users on a large public cloud for over three years. Our production results show that Dorado can accommodate up to 16M session entries, reduce memory cost by 80%, while enabling 50Mpps line-rate processing.
Heng Yu, Kai Ren, Jiajun Liang et al.· Proceedings of the ACM SIGCO...· 0 citations
Modern cloud gateways have evolved to include diverse network functions and heterogeneous hardware, such as programmable switches and FPGAs, to handle increasing workloads and minimize forwarding latency. Existing network tracing tools, however, operate primarily at device granularity and cannot pinpoint which function on which hardware component causes packet losses or latency spikes. To bridge this gap, we present CubeTrace, a unified, function-level flow tracing system that enables microscopic tracing inside heterogeneous cloud gateways. CubeTrace standardizes tracing units as cubes across different hardware platforms, regardless of their varied underlying implementations, and operates at flow granularity for reliability reasons. This introduces a new tracing abstraction for heterogeneous gateways while maintaining low overhead. Moreover, the collected flow-cube data by CubeTrace can be decoded into packet-level representations and integrated with well-established distributed tracing frameworks, enabling the use of off-the-shelf analysis tools. Our evaluations demonstrate that CubeTrace introduces minimal overhead, consuming less than 1% of memory resources and adding less than 1% to forwarding latency. Having been deployed in a large-scale cloud gateway, CubeTrace has significantly improved problem localization, reducing resolution times from hours or even days to just minutes.
Yunming Xiao, Yinchao Yang, Jiaqi Zheng et al.· Proceedings of the ACM SIGCO...· 1 citation
The experience in designing, deploying, and operating Pegasus, a data center network tailored for the AI cloud, along with operational lessons learned from its deployment are shared.
Xianneng Zou, Yadong Liu, Yiran Zhang et al.· Proceedings of the ACM SIGCO...· 0 citations
XFir is the first hardware-accelerated new-flow setup system for cloud host servers that delivers high CPS throughput while preserving sufficient flexibility, and is cost-effective to deploy, requiring only a single DPU per host.
Shihan Lin, Shunqiao Jiang, Liang Wang et al.· Proceedings of the ACM SIGCO...· 1 citation