#machine learning
Jun 2025
NoLoCo: No-all-reduce Low Communication Training Method for Large Models
A novel optimization method is proposed, NoLoCo, that does not explicitly synchronize all model parameters during training and does not require any collective communication, thereby reducing accelerator idle time and is more communication-efficient than DiLoCo.
J. Kolehmainen, Nikolay Blagoev, Semih Kara et al.
· arXiv.org · 5 citations