Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment
This work finds that utilizing chat templates renders the model more compliant with harmful queries compared to a non-chat template, and shows that using a non-chat template during distillation better preserves the base student's internal representations, while chat template distillation induces a larger representation...