الملخص
Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.
الكلمات المفتاحية
الموضوع
بيانات النشر
- المجلة
- غير متاح
- وصول مفتوح
- وصول مفتوح أخضر
اقتبس هذه المقالة
APA 7
Jiang, H., Zheng, M., Song, M., Zhong, Q., Sun, J., Zhang, T., & Fang, J. (2026). ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery. https://omanscience.com/ar/articles/recal-calibrating-structured-pruning-for-on-policy-distillation-recovery
MLA 9
Jiang, Houcheng, et al. "ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery." https://omanscience.com/ar/articles/recal-calibrating-structured-pruning-for-on-policy-distillation-recovery.
شيكاغو (المؤلف–التاريخ)
Jiang, Houcheng, Mao Zheng, Mingyang Song, Qiyong Zhong, Jie Sun, Tianyu Zhang, and Junfeng Fang. 2026. "ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery." https://omanscience.com/ar/articles/recal-calibrating-structured-pruning-for-on-policy-distillation-recovery.
هارفارد
Jiang, H., Zheng, M., Song, M., Zhong, Q., Sun, J., Zhang, T. and Fang, J. (2026) 'ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery', Available at: https://omanscience.com/ar/articles/recal-calibrating-structured-pruning-for-on-policy-distillation-recovery.
فانكوفر
Jiang H, Zheng M, Song M, Zhong Q, Sun J, Zhang T, et al. ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery. https://omanscience.com/ar/articles/recal-calibrating-structured-pruning-for-on-policy-distillation-recovery
IEEE
H. Jiang, M. Zheng, M. Song, Q. Zhong, J. Sun, T. Zhang, and J. Fang, "ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery," https://omanscience.com/ar/articles/recal-calibrating-structured-pruning-for-on-policy-distillation-recovery.