FabricPerf: Measuring NIC-less Scale-Up Network through GPU Communication Kernel Profiling
TL;DR
This work implements FabricPerf for the CUDA/NvLink ecosystem and validate its high precision and low overhead via extensive experiments on H100 and GB200 NVL72, and believes FabricPerf improves Scale-up network observability, laying a foundation for more complete and efficient network stacks.
Abstract
The rapid growth of distributed machine learning systems shifts ML networks from the CPU/GPU-NIC to GPU-Fabrics architectures, creating an emerging need to measure and optimize NIC-less Scale-up networks. Yet, the disappearance of NICs removes key observability points and leaves existing measurement tools largely limited to coarse, application-layer telemetry. We present FabricPerf, a Scale-up network measurement toolkit that enables fine-grained packet-level timestamping and cross-layer hardware insights. FabricPerf uses a software-hardware co-design that integrates protocol-level probing and GPTP clock synchronization within GPU communication kernels with memory profiling of physical pipelines. We implement FabricPerf for the CUDA/NvLink ecosystem and validate its high precision and low overhead via extensive experiments on H100 and GB200 NVL72. Using FabricPerf, we explore Scale-up network dynamics under varied configurations, with two case studies that reveal critical bottlenecks: (i) channel imbalance in high P99 latency and (ii) memory pipeline inefficiency of ~0% LLC hit-rate. To mitigate these issues, we leverage work-stealing to rebalance channels by 46% and boost throughput by 17.5GB/s, and optimize LLC eviction priorities for memory flows to elevate the LLC read hit-rate to ~35%, yielding 16.07GB/s bandwidth improvement. We believe FabricPerf improves Scale-up network observability, laying a foundation for more complete and efficient network stacks. FabricPerf is open-sourced at github.com/open-neutrino/fabricperf.