On-policy distillation (OPD) improves reasoning by providing token-level supervision from a teacher on a student's own trajectories. Existing methods primarily focus on enhancing this teacher-side guidance (e.g., by enriching teacher inputs and refining teacher feedback), yet we find that limited student perception is...
TurnWise is proposed, a turn-level GRPO framework that aligns learning with individual streaming decisions and addresses the mismatch between trajectory-level feedback and turn-level interaction, promoting informative responses and reducing redundancy in streaming video understanding.
Shuai Dong, Kang-Hui Tian, Ling-Han Chen et al.· 0 citations
The results support OPD as preparation for RL and favor forward KL when OPD is followed by RL in the authors' comparison, which highlights the importance of both objective choice and the states receiving supervision for subsequent RL.
This work compares the student view and training fixed within each scale against three abstractions compiled offline, a named strategy, a method-independent framing, and a problem category, and against an answer-only control that keeps the destination but removes the path.
Kang-Hui Tian, Si-Yuan Liu, Tianxiang Jiang et al.· 2 citations
Value Flattening is identified as an important yet overlooked failure mode of critic learning in standard PPO and a simple sparse supervision strategy can mitigate it; SParse Proximal Policy Optimization is introduced, which applies the value loss to only a few well-separated states in each response to mitigate both ef...
Yi-Zhuo Li, Jian-Hao Yan, Yun Luo et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.