Abstract
The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization into Muon to balance update magnitudes and improve pretraining performance. However, row-wise normalization alone cannot accommodate different imbalance patterns in update matrices. In this paper, we propose an improved Muon optimizer, called \underline{m}atrix-\underline{eq}uilibrating Muon~(MeqMuon), for LLM pretraining. MeqMuon balances both row and column magnitudes through normalization that can be automatically tailored to different imbalance patterns without manual intervention. Moreover, MeqMuon eliminates the need to store AdamW's second-moment estimates, reducing optimizer-state memory usage. Empirical results demonstrate that MeqMuon achieves better convergence performance than AdamW, Muon, and other baselines in LLM pretraining.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Shi, C. W., Wang, X., & Li, W. J. (2026). MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining. https://omanscience.com/en/articles/meqmuon-matrix-equilibrating-muon-for-llm-pretraining
MLA 9
Shi, Chang-Wei, et al. "MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining." https://omanscience.com/en/articles/meqmuon-matrix-equilibrating-muon-for-llm-pretraining.
Chicago (author–date)
Shi, Chang-Wei, Xu Wang, and Wu-Jun Li. 2026. "MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining." https://omanscience.com/en/articles/meqmuon-matrix-equilibrating-muon-for-llm-pretraining.
Harvard
Shi, C. W., Wang, X. and Li, W. J. (2026) 'MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining', Available at: https://omanscience.com/en/articles/meqmuon-matrix-equilibrating-muon-for-llm-pretraining.
Vancouver
Shi CW, Wang X, Li WJ. MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining. https://omanscience.com/en/articles/meqmuon-matrix-equilibrating-muon-for-llm-pretraining
IEEE
C. W. Shi, X. Wang, and W. J. Li, "MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining," https://omanscience.com/en/articles/meqmuon-matrix-equilibrating-muon-for-llm-pretraining.