Abstract
Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a tension behind this result: concentrated spectra can suppress gradient directions in coupled weight matrices and slow optimization, while constraints maintained throughout training can limit task-specific adaptation and raise the attainable loss floor. We introduce ORCA (Orthogonal Regularization, Cooled After), a minimal optimizer intervention that applies strong but temporary soft orthogonality regularization early in training, then removes it. This allows the weights to benefit from a broader spectrum early on and adapt freely afterward. Across LLaMA, Qwen3, and fine-grained mixture-of-experts models ranging from 130M to 8B parameters, ORCA achieves lower final validation loss than Muon. Its loss reduction relative to Muon matches or exceeds Muon's reduction relative to Adam. Ablations support the early-shaping, later-release design. Further, ORCA requires no architectural changes and adds minimal overhead.
Keywords
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Liu, Y., Jiang, B., Hou, L., Tao, X., Wan, P., Lin, Z., & Fang, C. (2026). ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training. https://omanscience.com/en/articles/orca-the-annealed-spectral-conditioning-optimizer-for-faster-better-llm-training
MLA 9
Liu, Yuanshi, et al. "ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training." https://omanscience.com/en/articles/orca-the-annealed-spectral-conditioning-optimizer-for-faster-better-llm-training.
Chicago (author–date)
Liu, Yuanshi, Boyuan Jiang, Liang Hou, Xin Tao, Pengfei Wan, Zhouchen Lin, and Cong Fang. 2026. "ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training." https://omanscience.com/en/articles/orca-the-annealed-spectral-conditioning-optimizer-for-faster-better-llm-training.
Harvard
Liu, Y., Jiang, B., Hou, L., Tao, X., Wan, P., Lin, Z. and Fang, C. (2026) 'ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training', Available at: https://omanscience.com/en/articles/orca-the-annealed-spectral-conditioning-optimizer-for-faster-better-llm-training.
Vancouver
Liu Y, Jiang B, Hou L, Tao X, Wan P, Lin Z, et al. ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training. https://omanscience.com/en/articles/orca-the-annealed-spectral-conditioning-optimizer-for-faster-better-llm-training
IEEE
Y. Liu, B. Jiang, L. Hou, X. Tao, P. Wan, Z. Lin, and C. Fang, "ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training," https://omanscience.com/en/articles/orca-the-annealed-spectral-conditioning-optimizer-for-faster-better-llm-training.