Data-free On-policy Distillation
DF-OPD matches and even surpasses real data, and the questions it produces track the teacher's own post-training data on three key diagnostics of training dynamics, which other real datasets do not.
Gengsheng Li, Mao Zheng, Ming-Yang Song et al.
· 1 citation