Christian Wagner

Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

This work proposes a student design based on simple, homogeneous blocks mirroring those of the teacher, distilling knowledge between corresponding blocks, showing that intermediate block-wise distillation, guided appropriately, is key to building compact data-efficient models without sacrificing accuracy.