R. Cohen

DualColl: Throughput-Optimal Bandwidth-Bound Collective Communications via Fractional Tree Packing

Large-scale distributed training increasingly makes collective communication bandwidth-bound. In modern GPU clusters, communication libraries such as NCCL implement collectives including AllReduce over heterogeneous interconnects such as NVLink, InfiniBand, and Ethernet, making efficient use of available network bandwi...