Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
Rollout Routing Replay (R3), a method that records routing distributions from the inference engine and replays them during training significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed, is proposed.