How Inner Optimizers Shape Replica Scaling
Training large language models (LLMs) across geographically distributed and poorly interconnected compute pools or data centers has become a topic of significant interest. This has led to increased study of local optimization methods such as DiLoco for LLMs, especially those that use local (or inner) adaptive or precon...
Eugene Belilovsky, Stefan Horoi, Edouard Oyallon et al.
· 0 citations