Abstract
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Chen, X., Su, F., Wang, K., Chen, J., & Li, T. (2026). Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation. https://omanscience.com/en/articles/spend-teacher-tokens-where-they-matter-success-referenced-on-policy-distillation
MLA 9
Chen, Xiang, et al. "Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation." https://omanscience.com/en/articles/spend-teacher-tokens-where-they-matter-success-referenced-on-policy-distillation.
Chicago (author–date)
Chen, Xiang, Futao Su, Kong Wang, Jiayi Chen, and TanLin Li. 2026. "Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation." https://omanscience.com/en/articles/spend-teacher-tokens-where-they-matter-success-referenced-on-policy-distillation.
Harvard
Chen, X., Su, F., Wang, K., Chen, J. and Li, T. (2026) 'Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation', Available at: https://omanscience.com/en/articles/spend-teacher-tokens-where-they-matter-success-referenced-on-policy-distillation.
Vancouver
Chen X, Su F, Wang K, Chen J, Li T. Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation. https://omanscience.com/en/articles/spend-teacher-tokens-where-they-matter-success-referenced-on-policy-distillation
IEEE
X. Chen, F. Su, K. Wang, J. Chen, and T. Li, "Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation," https://omanscience.com/en/articles/spend-teacher-tokens-where-they-matter-success-referenced-on-policy-distillation.