It is found that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.
Abstract
On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in English, Korean, and Japanese. OPD$^2$ improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD$^2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.
WDL-OPD is introduced, a mixture-constrained co-training method with two trainable policies that shows that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$^2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.
Ze-Hao Chen, Gong-Xun Li, Tianxiang Ai et al.· 0 citations
Contrastive Reinforced Policy Optimization (CRPO) is introduced, which reformulates agentic OPSD from a contrastive learning perspective, and conducts group-wise contrast to preserve reliable, fine-grained optimization signals.
Xingjian Wu, Junlin Liu, Xing-Chen Liu et al.· arXiv.org· 2 citations
Tail-Aware Top-$k$ OPD is proposed, a novel distillation method that restores the missing tail probability signal and better aligns the student's next-token distribution with the teacher's, preventing the increase in tail probability and entropy caused by top-$k$ normalization.
Flux-OPD is proposed, an OPD paradigm that uses evolving contexts as in-training supervision to capture task preferences in open-ended domains and outperforms existing OPD paradigms, highlighting the potential to combine teacher supervision with evolving contexts.
It is shown that a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks.
Bo-Yang Li, Bingsen Chen, Cheng-Hao Yang et al.· 11 citations
This work introduces $\beta$-OPSD and derives its optimal policy as a geometric interpolation between the reference policy and the privileged teacher, and provides a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical.
