Trajectory Dropout provides a simple mechanism for strengthening token-level supervision across model scales and OPD objectives and consistently improves average performance across teacher--student model pairs of different scales and six mathematical reasoning benchmarks.
Abstract
On-policy distillation (OPD) trains a student model on its own trajectories using dense token-level feedback from a stronger teacher model. Since each update is conditioned on the reasoning prefix already generated by the student, the prefix also shapes how effectively teacher feedback is converted into learning. We find that shared prefixes can lead to weak token-level updates, a phenomenon we call Prefix-Induced Supervision Attenuation (PISA). This attenuation arises in two common cases. (i) High student confidence can weaken corrective gradients even when the teacher disagrees. (ii) Tokens that rely on earlier reasoning can receive learning signals as weak as those for simple local continuations. To solve this problem, we propose Trajectory Dropout, a simple training-time intervention that exposes these weakened signals. The student first performs a standard full-context rollout to generate a complete trajectory. During training, we randomly drop a certain proportion of the student's reasoning trajectory, while the teacher continues to observe the complete trajectory for token-level supervision. This intervention strengthens corrections for overconfident predictions and introduces additional supervision at prefix-sensitive positions. Trajectory Dropout consistently improves average performance across teacher--student model pairs of different scales and six mathematical reasoning benchmarks, while also yielding gains on two out-of-domain benchmarks. It can also be flexibly integrated into existing OPD variants with negligible computational overhead, further improving their performance. These results demonstrate that Trajectory Dropout provides a simple mechanism for strengthening token-level supervision across model scales and OPD objectives.
This work uses teacher-successful problems to define a cheap reference for what the student can learn and measures how the likelihood of each observed token in trajectories from teacher-failed problems changes as an operational learnability signal, which can be computed once from stored trajectories and model checkpoin...
This work introduces SAKI (Supervision Allocation with KL-constrained Interpolation), which combines a KL-constrained teacher-guided rollout with maximal coupling and reuses realized accept/correction events to route token-level supervision.
Miteto Wei, Xiao-Han Wang, Ze-Hao Chen et al.· 0 citations
DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks, and its comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks.
ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison.
Wan-Qi Ren, Jian-Xiang Wang, Dan-Xuan Liu et al.· arXiv.org· 0 citations
A simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD), which forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher, support teacher-guided branching as a way to expose...
Taeckyung Lee, Rinat Amankos, Jeonghye Kim et al.· arXiv.org· 1 citation
This work proposes Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout and designs a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and proce...
Shu-Tong Wu, Xi-Wen Chen, B. Rappazzo et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.