Karl Ni

Beyond frozen teachers: co-training and co-distillation for task-specific fine-tuning

Across the seven tasks and hyperparameter configurations the authors consider, CTCD is competitive with vanilla distillation and outperforms supervised fine-tuning alone on average; it transfers to task-specific fine-tuning as a viable distillation objective, but not as an improvement over the standard formulation.