LastOPD: Taming Collapse in Latent On-Policy Distillation
LastOPD is proposed, which applies the latent signal only at the last-layer state, the common interface both LM heads read, and only during a 10-step crossfade into token-level OPD, which keeps the useful part of the latent signal and hands the student to token-level supervision before the collapse sets in.
Jie Yang, Zheng-Yu Fang, Ze-Lin Xu et al.
· 2 citations
· ⚡1