Abstract
On-policy distillation (OPD) uses teacher correction on student-generated responses. Full-vocabulary correction can provide important corrections even for tokens that the student assigns low probability, but backpropagating through all token logits becomes memory-intensive for long sequences. Existing memory-saving approaches estimate corrections from sampled tokens or restrict supervision to the student's TopK tokens, introducing sampling noise or changing the full-vocabulary correction. We introduce \textbf{SparseOPD}, which uses full-vocabulary teacher correction to determine which corrections matter before selecting the token logits to differentiate. SparseOPD first constructs the full-vocabulary correction without retaining its backward graph, then selects tokens by correction magnitude rather than student probability. Signed residual compensation preserves the total promoting and suppressing correction mass, while correction-aware budget allocation distributes the sparse support across positions. Finally, the update backpropagates only through the selected token logits. Across six task--scale settings spanning mathematics, chemistry QA, and multimodal reasoning, SparseOPD outperforms Sampled Token and TopK in task-average accuracy and matches or exceeds Full Vocabulary. Gradient cosine similarity reaches 99\% on 4B mathematics, while 8K full-parameter profiling shows 70.5\% lower backward memory.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Miao, Y., Liu, S., Liu, Y., Bai, Y., & Du, M. (2026). Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation. https://omanscience.com/en/articles/look-before-you-select-rethinking-vocabulary-sparsification-in-on-policy-distillation
MLA 9
Miao, Yongliang, et al. "Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation." https://omanscience.com/en/articles/look-before-you-select-rethinking-vocabulary-sparsification-in-on-policy-distillation.
Chicago (author–date)
Miao, Yongliang, Shuang Liu, Yanguang Liu, Yandong Bai, and Mengnan Du. 2026. "Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation." https://omanscience.com/en/articles/look-before-you-select-rethinking-vocabulary-sparsification-in-on-policy-distillation.
Harvard
Miao, Y., Liu, S., Liu, Y., Bai, Y. and Du, M. (2026) 'Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation', Available at: https://omanscience.com/en/articles/look-before-you-select-rethinking-vocabulary-sparsification-in-on-policy-distillation.
Vancouver
Miao Y, Liu S, Liu Y, Bai Y, Du M. Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation. https://omanscience.com/en/articles/look-before-you-select-rethinking-vocabulary-sparsification-in-on-policy-distillation
IEEE
Y. Miao, S. Liu, Y. Liu, Y. Bai, and M. Du, "Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation," https://omanscience.com/en/articles/look-before-you-select-rethinking-vocabulary-sparsification-in-on-policy-distillation.