Beyond frozen teachers: co-training and co-distillation for task-specific fine-tuning

Abstract

Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student model, facilitating deployment in resource-constrained environments while minimizing performance degradation. The dominant KD formulation assumes a frozen teacher, already trained on the task, that supervises the student through its terminal output distribution. Co-Training and Co-Distillation (CTCD), in contrast, jointly trains teacher and student under bidirectional soft-target losses. This study investigates whether the advantage reported for CTCD during pretraining persists under task-specific fine-tuning. To answer this question, we conduct a six-part ablation study across seven GLUE tasks, using BERT-base as the teacher and DistilBERT as the student. CTCD produces the best student on three of the seven tasks, but vanilla distillation achieves the strongest average student performance, and seed-to-seed variation is large enough to preclude a conclusive student-side ranking. The clearest effect of joint training appears on the teacher rather than the student: relative to otherwise one-way co-training, full CTCD improves the teacher mean on all seven tasks. Interestingly, this advantage does not translate into a corresponding student gain. To probe the mechanism, we track teacher entropy and adaptive calibration error (ACE) along the joint learning trajectory. CTCD does exhibit the trajectory-level signals that prior work associates with effective distillation, yet the resulting spread in student performance across ablations remains within seed variance. Across the seven tasks and hyperparameter configurations we consider, CTCD is competitive with vanilla distillation and outperforms supervised fine-tuning alone on average; it therefore transfers to task-specific fine-tuning as a viable distillation objective, but not as an improvement over the standard formulation.