On-Policy Transition Distillation
On-policy distillation (OPD) consolidates expert capabilities into a single student by matching teacher and student final-layer distributions on student-generated rollouts. We investigate whether depthwise prediction transitions, the changes in predictions between layers, can provide additional task-relevant supervisio...
Run-Yang You, Yong-Qi Li, Tie-Zheng Yu et al.
· 0 citations