Abstract
Prior research has demonstrated that the choice of prompt template during Supervised Fine-Tuning (SFT) significantly impacts the robustness of safety alignment afterwards. However, the influence of template selection during Knowledge Distillation (KD) from teacher to student remains largely unexplored. Thus, we fill this gap by analyzing how different template configurations influence the pre-existing safety alignment of the student. We observe a significant degradation of safety alignment present in the aligned base instruct-tuned model. Specifically, we find that utilizing chat templates renders the model more compliant with harmful queries compared to a non-chat template. These findings are consistent across three models: LLaMA, Gemma and Qwen model families and are evaluated across multiple safety benchmarks. We further show that using a non-chat template during distillation better preserves the base student's internal representations, while chat template distillation induces a larger representational shift. Code: https://github.com/anjilab/role-of-prompt-template-in-kd
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Budathoki, A., Dhakal, M., Ampel, B. M., & Ding, Y. (2026). Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment. https://omanscience.com/en/articles/understanding-the-role-of-prompt-template-in-knowledge-distillation-for-safety-alignment
MLA 9
Budathoki, Anjila, et al. "Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment." https://omanscience.com/en/articles/understanding-the-role-of-prompt-template-in-knowledge-distillation-for-safety-alignment.
Chicago (author–date)
Budathoki, Anjila, Manish Dhakal, Benjamin M. Ampel, and Yi Ding. 2026. "Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment." https://omanscience.com/en/articles/understanding-the-role-of-prompt-template-in-knowledge-distillation-for-safety-alignment.
Harvard
Budathoki, A., Dhakal, M., Ampel, B. M. and Ding, Y. (2026) 'Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment', Available at: https://omanscience.com/en/articles/understanding-the-role-of-prompt-template-in-knowledge-distillation-for-safety-alignment.
Vancouver
Budathoki A, Dhakal M, Ampel BM, Ding Y. Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment. https://omanscience.com/en/articles/understanding-the-role-of-prompt-template-in-knowledge-distillation-for-safety-alignment
IEEE
A. Budathoki, M. Dhakal, B. M. Ampel, and Y. Ding, "Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment," https://omanscience.com/en/articles/understanding-the-role-of-prompt-template-in-knowledge-distillation-for-safety-alignment.