Preprint Open access
Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State Quantization
Quantizing AdamW's optimizer states reduces persistent storage, but quantization errors propagate through the moment recurrences and perturb subsequent adaptive updates. We redesign 4-bit optimizer-state quantization for AdamW from the perspective of \emph{rounding space}: the coordinate in which a quantizer chooses be …