MoECatch: Efficient Checkpointing and Consistent Recovery for Mixture-of-Experts Models
Checkpointing is essential for fault-tolerant large language model training, but it increasingly limits persistence frequency and recovery efficiency as training runs become longer and clusters become larger. This problem is especially acute in Mixture-of-Experts (MoE) systems, where checkpoint bytes are dominated by e...