Abstract
Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning. However, they face a distinct credit-assignment challenge, since a few commitments during denoising sharply reduce the uncertainty over the remaining masked positions and shape much of the response. Most post-training recipes for dLMs do not use this signal to decide which tokens to train on: they typically train on the final text or assign rewards to whole denoising steps, rather than selecting the individual commitments that shape the response. We introduce Pivot-SD, an efficient offline self-distillation framework that supervises only these high-impact commitments (pivots). Pivot-SD selects pivots using an information-gain metric measuring uncertainty reduction over the remaining masked positions. Pivots from successful trajectories are trained with cross-entropy, and pivots from failed trajectories with targeted unlikelihood, leaving the rest of the failed trajectory untouched. Using only 200 questions and four rollouts each, Pivot-SD improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Kim, S. H., Hong, S., Choi, Y., Chao, C. H., Yun, S. Y., & Krishnan, R. G. (2026). Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models. https://omanscience.com/en/articles/pivot-sd-efficient-self-distillation-for-masked-diffusion-language-models
MLA 9
Kim, Seo Hyun, et al. "Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models." https://omanscience.com/en/articles/pivot-sd-efficient-self-distillation-for-masked-diffusion-language-models.
Chicago (author–date)
Kim, Seo Hyun, Sunwoo Hong, Younwoo Choi, Chen-Hao Chao, Se-Young Yun, and Rahul G. Krishnan. 2026. "Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models." https://omanscience.com/en/articles/pivot-sd-efficient-self-distillation-for-masked-diffusion-language-models.
Harvard
Kim, S. H., Hong, S., Choi, Y., Chao, C. H., Yun, S. Y. and Krishnan, R. G. (2026) 'Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models', Available at: https://omanscience.com/en/articles/pivot-sd-efficient-self-distillation-for-masked-diffusion-language-models.
Vancouver
Kim SH, Hong S, Choi Y, Chao CH, Yun SY, Krishnan RG. Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models. https://omanscience.com/en/articles/pivot-sd-efficient-self-distillation-for-masked-diffusion-language-models
IEEE
S. H. Kim, S. Hong, Y. Choi, C. H. Chao, S. Y. Yun, and R. G. Krishnan, "Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models," https://omanscience.com/en/articles/pivot-sd-efficient-self-distillation-for-masked-diffusion-language-models.