No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation
Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to re...
Seong-Jang Kim, Chaeyun Jang, Noah Lee et al.
· 0 citations