Skip to content
Preprint

What Shared Prefixes Hide: Trajectory Dropout for On-Policy Distillation

Sep 2026 · 0 citations · 29 references
Computer Science

TL;DR

Trajectory Dropout provides a simple mechanism for strengthening token-level supervision across model scales and OPD objectives and consistently improves average performance across teacher--student model pairs of different scales and six mathematical reasoning benchmarks.

Abstract

On-policy distillation (OPD) trains a student model on its own trajectories using dense token-level feedback from a stronger teacher model. Since each update is conditioned on the reasoning prefix already generated by the student, the prefix also shapes how effectively teacher feedback is converted into learning. We find that shared prefixes can lead to weak token-level updates, a phenomenon we call Prefix-Induced Supervision Attenuation (PISA). This attenuation arises in two common cases. (i) High student confidence can weaken corrective gradients even when the teacher disagrees. (ii) Tokens that rely on earlier reasoning can receive learning signals as weak as those for simple local continuations. To solve this problem, we propose Trajectory Dropout, a simple training-time intervention that exposes these weakened signals. The student first performs a standard full-context rollout to generate a complete trajectory. During training, we randomly drop a certain proportion of the student's reasoning trajectory, while the teacher continues to observe the complete trajectory for token-level supervision. This intervention strengthens corrections for overconfident predictions and introduces additional supervision at prefix-sensitive positions. Trajectory Dropout consistently improves average performance across teacher--student model pairs of different scales and six mathematical reasoning benchmarks, while also yielding gains on two out-of-domain benchmarks. It can also be flexibly integrated into existing OPD variants with negligible computational overhead, further improving their performance. These results demonstrate that Trajectory Dropout provides a simple mechanism for strengthening token-level supervision across model scales and OPD objectives.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers

This work uses teacher-successful problems to define a cheap reference for what the student can learn and measures how the likelihood of each observed token in trajectories from teacher-failed problems changes as an operational learnability signal, which can be computed once from stored trajectories and model checkpoin...

Yi-Hao Ai, Wei-Long Yan · 0 citations

Recursive Self-Improvement via On-Policy Distillation for Reasoning

DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks, and its comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks.

Shang-Jian Yin, Ze-Hao Zhao, Kavosh Asadi et al. · 6 citations

Learning from Think-Mode Advantage via On-Policy Distillation

ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison.

Wan-Qi Ren, Jian-Xiang Wang, Dan-Xuan Liu et al. · 0 citations

TISD: On-Policy Self-Distillation with Trajectory Intervention

A simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD), which forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher, support teacher-guided branching as a way to expose...

Taeckyung Lee, Rinat Amankos, Jeonghye Kim et al. · 1 citation
#artificial intelligence Preprint Sep 2026

Interactive-Policy Distillation with Bidirectional Propose-and-Verify

This work proposes Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout and designs a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and proce...

Shu-Tong Wu, Xi-Wen Chen, B. Rappazzo et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.