Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
This work proposes a curvature-conditioned multiscale momentum method with sphere constraints, which significantly accelerates Muon across diverse architectures (dense, MoE) and model sizes (0.12B--2.3B parameters).
Shu-Chen Zhu, Yu-Xin Fang, Ming-Ze Wang et al.
· 0 citations