Preprint Open access
Muon Learns Facts Better: Understanding the Role of Spectral Orthogonalization
The Muon optimizer applies spectral orthogonalization to matrix-valued updates and has shown strong performance in large-scale neural network training, yet the mechanisms of this transformation in feature learning remain poorly understood. In this work, we investigate this question through a tractable factual-recall mo …