MoECatch: Efficient Checkpointing and Consistent Recovery for Mixture-of-Experts Models
Abstract
Checkpointing is essential for fault-tolerant large language model training, but it increasingly limits persistence frequency and recovery efficiency as training runs become longer and clusters become larger. This problem is especially acute in Mixture-of-Experts (MoE) systems, where checkpoint bytes are dominated by expert parameters and optimizer states. Conventional checkpointing persists all experts uniformly at every checkpoint, even though expert updates are highly uneven under skewed token routing. This mismatch wastes checkpointing bandwidth on persisting “barely changed experts”, limits attainable checkpoint frequency, and increases rollback cost after failures. To address this mismatch, we present MoECatch, a sparse checkpointing system for MoE training that treats sparse checkpointing as a coupled checkpoint-and-recovery problem. On the checkpoint path, Aging-then-Greedy Expert Selection (AGES) prioritizes which experts to persist under a fixed budget. On the recovery path, Version Catch-Up (VCU) reconstructs stale experts from mixed-version checkpoints while leaving already aligned state unchanged. MoECatch further accelerates recovery through minimum executable-state restoration and asynchronous optimizer staging. We implement MoECatch in the MindSpeed framework. The results show that MoECatch improves attainable checkpoint frequency by up to 2.5 × and reduces end-to-end wasted time after failures by up to \(22.86\%\). Across both pre-training and fine-tuning workloads, recovery preserves average downstream-task accuracy relative to full-checkpoint recovery. At the state level, the relative L2 parameter deviation between the recovered model and the same-iteration full checkpoint does not exceed \(0.001\%\).