Preprint Open access
ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a tension behind this result: concentrated spectra can suppress gradient directions in coupled weight matrices and slow optimization, while con …