Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on domain labels that mixed training corpora often lack and cannot...
Jie Sun, Mao Zheng, Ming-Yang Song et al.
· 3 citations