Abstract
Discrete diffusion language models can generate multiple tokens in parallel, but reducing the number of denoising steps can lead to inconsistent predictions. Standard cross-entropy training fits conditional token marginals, whereas parallel generation requires consistent joint predictions. We introduce Alpha Diffusion Language Models (AlphaDLM), trained with a sequence-level alpha loss that recovers cross-entropy in the limit of vanishing alpha and has a joint-mode optimum at alpha one. Our analysis characterizes how the objective and factorization jointly determine the fitted distribution. We identify conditions under which intermediate alpha preserves multiple valid completions while excluding invalid token combinations. Trained on TinyGSM, our method achieves 34.6% accuracy on GSM8K with only four model evaluations. We further scale the method to SDAR-1.7B and evaluate it on code and mathematics benchmarks. These results show that changing the training objective can improve the accuracy-computation trade-off of factorized diffusion language models.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Gushchin, N., Baranchuk, D., & Korotin, A. (2026). Alpha Diffusion Language Models: Factorization Alone Is Not the Problem. https://omanscience.com/en/articles/alpha-diffusion-language-models-factorization-alone-is-not-the-problem
MLA 9
Gushchin, Nikita, et al. "Alpha Diffusion Language Models: Factorization Alone Is Not the Problem." https://omanscience.com/en/articles/alpha-diffusion-language-models-factorization-alone-is-not-the-problem.
Chicago (author–date)
Gushchin, Nikita, Dmitry Baranchuk, and Alexander Korotin. 2026. "Alpha Diffusion Language Models: Factorization Alone Is Not the Problem." https://omanscience.com/en/articles/alpha-diffusion-language-models-factorization-alone-is-not-the-problem.
Harvard
Gushchin, N., Baranchuk, D. and Korotin, A. (2026) 'Alpha Diffusion Language Models: Factorization Alone Is Not the Problem', Available at: https://omanscience.com/en/articles/alpha-diffusion-language-models-factorization-alone-is-not-the-problem.
Vancouver
Gushchin N, Baranchuk D, Korotin A. Alpha Diffusion Language Models: Factorization Alone Is Not the Problem. https://omanscience.com/en/articles/alpha-diffusion-language-models-factorization-alone-is-not-the-problem
IEEE
N. Gushchin, D. Baranchuk, and A. Korotin, "Alpha Diffusion Language Models: Factorization Alone Is Not the Problem," https://omanscience.com/en/articles/alpha-diffusion-language-models-factorization-alone-is-not-the-problem.