Matching Supervision to the Student's Learning Capacity: A Unified Framework for On-Policy Self-Distillation
Unified On-Policy Self-Distillation (USD), a lightweight online algorithm to solve the Lagrangian, reveals that a single dual variable governs both decisions: at one price for learning difficulty, it simultaneously sets the token-selection threshold and the direction of privileged-information adjustment, keeping superv...