Abstract
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Zhao, S., Fu, Y., Jiang, J., Liu, S. Y., Bian, S., Lee, B. K., Sreenivas, S. T., Dai, W., Ye, H., Grover, A., & Molchanov, P. (2026). When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better. https://omanscience.com/en/articles/when-do-we-need-on-policy-distillation-distilling-on-offline-student-rollouts-is-often-better
MLA 9
Zhao, Siyan, et al. "When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better." https://omanscience.com/en/articles/when-do-we-need-on-policy-distillation-distilling-on-offline-student-rollouts-is-often-better.
Chicago (author–date)
Zhao, Siyan, Yonggan Fu, Jindong Jiang, Shih-Yang Liu, Song Bian, Byung-Kwan Lee, Sharath Turuvekere Sreenivas, Wenliang Dai, Hanrong Ye, Aditya Grover, and Pavlo Molchanov. 2026. "When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better." https://omanscience.com/en/articles/when-do-we-need-on-policy-distillation-distilling-on-offline-student-rollouts-is-often-better.
Harvard
Zhao, S., Fu, Y., Jiang, J., Liu, S. Y., Bian, S., Lee, B. K., Sreenivas, S. T., Dai, W., Ye, H., Grover, A. and Molchanov, P. (2026) 'When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better', Available at: https://omanscience.com/en/articles/when-do-we-need-on-policy-distillation-distilling-on-offline-student-rollouts-is-often-better.
Vancouver
Zhao S, Fu Y, Jiang J, Liu SY, Bian S, Lee BK, et al. When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better. https://omanscience.com/en/articles/when-do-we-need-on-policy-distillation-distilling-on-offline-student-rollouts-is-often-better
IEEE
S. Zhao, Y. Fu, J. Jiang, S. Y. Liu, S. Bian, B. K. Lee, S. T. Sreenivas, W. Dai, H. Ye, A. Grover, and P. Molchanov, "When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better," https://omanscience.com/en/articles/when-do-we-need-on-policy-distillation-distilling-on-offline-student-rollouts-is-often-better.