Rollout Routing Replay (R3), a method that records routing distributions from the inference engine and replays them during training significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed, is proposed.
Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and adherence to task re...
Jin-Hao Dong, Liang Zhao, Zi-Hao Yue et al.· 1 citation
On an 80B-A3B MoE model, HySparse2 outperforms HySparse and Hybrid SWA on long-context retrieval and multi-turn agentic tasks, while substantially reducing prefill computation and KV-cache storage.