Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs
Multi-Teacher Self-Distillation Policy Optimization is introduced, an on-policy distillation method that unifies several frozen teachers into one student model and lifts the weakest domain of Qwen3-8B by 14.79 points and narrows its domain gap by 74.7%, a better balance than serving one matched teacher per domain.
Xi-Xiang He, Xing-Ming Li, Bai-Qi Wu et al.
· 5 citations
· ⚡1