Self-Distillation with Verbal Regularization
On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distribut...
Jiazheng Li, Yawei Li, Jun-Ru Lu et al.
· 0 citations