Abstract
Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional ODE/SDE latent trajectories. Trained blockwise via likelihood optimization, Sigma jointly denoises Gaussian-corrupted token embeddings while learning an optimal embedding geometry. To accelerate training, Sigma leverages pre-trained weights from autoregressive (AR) models for warm-starting. During inference, we identify classifier-free guidance and score temperature as essential for high-fidelity reasoning and coding. Across comprehensive math reasoning and coding evaluations against state-of-the-art discrete counterparts (masked dLMs and AR baselines), Sigma achieves competitive performance with discrete models on standard benchmarks (e.g., GSM8K, Minerva, HumanEval, MBPP) after pre-training and on challenging reasoning tasks (e.g., MATH-500, AIME) after supervised fine-tuning. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: (i) embedding-space steering effectively governs the quality-diversity trade-off, yielding strong pass@k performance and (ii) continuous trajectories enable graceful degradation for low NFEs and efficient distillation. These establish continuous dLMs as a promising paradigm for efficient language generation.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Yang, Z., Guo, W., Lemercier, J. M., Welker, S., Fu, Y., Kamani, M. M., Norouzi, S., Berner, J., Geffner, T., Kreis, K., Chen, Y., Tao, M., Thickstun, J., Molchanov, P., Jukić, A., Vahdat, A., & Mardani, M. (2026). Large Language Continuous Diffusion Models. https://omanscience.com/en/articles/large-language-continuous-diffusion-models
MLA 9
Yang, Zhihan, et al. "Large Language Continuous Diffusion Models." https://omanscience.com/en/articles/large-language-continuous-diffusion-models.
Chicago (author–date)
Yang, Zhihan, Wei Guo, Jean-Marie Lemercier, Simon Welker, Yonggan Fu, Mohammad Mahdi Kamani, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Jukić, Arash Vahdat, and Morteza Mardani. 2026. "Large Language Continuous Diffusion Models." https://omanscience.com/en/articles/large-language-continuous-diffusion-models.
Harvard
Yang, Z., Guo, W., Lemercier, J. M., Welker, S., Fu, Y., Kamani, M. M., Norouzi, S., Berner, J., Geffner, T., Kreis, K., Chen, Y., Tao, M., Thickstun, J., Molchanov, P., Jukić, A., Vahdat, A. and Mardani, M. (2026) 'Large Language Continuous Diffusion Models', Available at: https://omanscience.com/en/articles/large-language-continuous-diffusion-models.
Vancouver
Yang Z, Guo W, Lemercier JM, Welker S, Fu Y, Kamani MM, et al. Large Language Continuous Diffusion Models. https://omanscience.com/en/articles/large-language-continuous-diffusion-models
IEEE
Z. Yang, W. Guo, J. M. Lemercier, S. Welker, Y. Fu, M. M. Kamani, S. Norouzi, J. Berner, T. Geffner, K. Kreis, Y. Chen, M. Tao, J. Thickstun, P. Molchanov, A. Jukić, A. Vahdat, and M. Mardani, "Large Language Continuous Diffusion Models," https://omanscience.com/en/articles/large-language-continuous-diffusion-models.