Abstract
Pretraining a language model takes enormous compute, and the right optimizer can save a good part of it. Muon's spectral step gives a stronger direction than a sign step, but it is expensive. Every step runs Newton-Schulz iterations on the full matrix and, in distributed training, an extra all-reduce. Sign steps, as in Lion and Signum, are cheap and stay local to each device. We propose LionMuon, which takes one Muon step every $P$ iterations and Lion steps in between, with a single dual-EMA momentum buffer shared by both. Muon's compute and communication are paid once per $P$ steps, and the optimizer state is half of AdamW's. A single-EMA variant, SignMuon, already improves on Muon. We prove complexity bounds under heavy-tailed noise in which the period sets an interpolation between Muon's and Lion's smoothness and noise constants, and which say when LionMuon is faster than both. On 124M and 355M models trained on FineWeb, LionMuon with $P=2$ and $P=5$ reaches a lower loss than Muon, AdamW, Lion and Signum at the same number of tokens. Under 4-GPU data-parallel training it reaches Muon's final loss with a third less wall-clock on PCIe, and it beats the communication-efficient Muon variants Dion and MuonBP on loss at no more exposed communication, while keeping the exact gradient. Code: https://github.com/brain-lab-research/lion-muon
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Bolatov, A., Riabinin, A., Kornilov, N., Veprikov, A., Horváth, S., Takáč, M., & Beznosikov, A. (2026). LionMuon: Alternating Spectral and Sign Descent for Efficient Training. https://omanscience.com/en/articles/lionmuon-alternating-spectral-and-sign-descent-for-efficient-training
MLA 9
Bolatov, Arman, et al. "LionMuon: Alternating Spectral and Sign Descent for Efficient Training." https://omanscience.com/en/articles/lionmuon-alternating-spectral-and-sign-descent-for-efficient-training.
Chicago (author–date)
Bolatov, Arman, Artem Riabinin, Nikita Kornilov, Andrey Veprikov, Samuel Horváth, Martin Takáč, and Aleksandr Beznosikov. 2026. "LionMuon: Alternating Spectral and Sign Descent for Efficient Training." https://omanscience.com/en/articles/lionmuon-alternating-spectral-and-sign-descent-for-efficient-training.
Harvard
Bolatov, A., Riabinin, A., Kornilov, N., Veprikov, A., Horváth, S., Takáč, M. and Beznosikov, A. (2026) 'LionMuon: Alternating Spectral and Sign Descent for Efficient Training', Available at: https://omanscience.com/en/articles/lionmuon-alternating-spectral-and-sign-descent-for-efficient-training.
Vancouver
Bolatov A, Riabinin A, Kornilov N, Veprikov A, Horváth S, Takáč M, et al. LionMuon: Alternating Spectral and Sign Descent for Efficient Training. https://omanscience.com/en/articles/lionmuon-alternating-spectral-and-sign-descent-for-efficient-training
IEEE
A. Bolatov, A. Riabinin, N. Kornilov, A. Veprikov, S. Horváth, M. Takáč, and A. Beznosikov, "LionMuon: Alternating Spectral and Sign Descent for Efficient Training," https://omanscience.com/en/articles/lionmuon-alternating-spectral-and-sign-descent-for-efficient-training.