Abstract
Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Ko, J., Parshakova, T., Cai, D., & Gower, R. M. (2026). SoftServe: A Scalable Quasi-Newton Method for Deep Learning. https://omanscience.com/en/articles/softserve-a-scalable-quasi-newton-method-for-deep-learning
MLA 9
Ko, Joohwan, et al. "SoftServe: A Scalable Quasi-Newton Method for Deep Learning." https://omanscience.com/en/articles/softserve-a-scalable-quasi-newton-method-for-deep-learning.
Chicago (author–date)
Ko, Joohwan, Tetiana Parshakova, Diana Cai, and Robert M. Gower. 2026. "SoftServe: A Scalable Quasi-Newton Method for Deep Learning." https://omanscience.com/en/articles/softserve-a-scalable-quasi-newton-method-for-deep-learning.
Harvard
Ko, J., Parshakova, T., Cai, D. and Gower, R. M. (2026) 'SoftServe: A Scalable Quasi-Newton Method for Deep Learning', Available at: https://omanscience.com/en/articles/softserve-a-scalable-quasi-newton-method-for-deep-learning.
Vancouver
Ko J, Parshakova T, Cai D, Gower RM. SoftServe: A Scalable Quasi-Newton Method for Deep Learning. https://omanscience.com/en/articles/softserve-a-scalable-quasi-newton-method-for-deep-learning
IEEE
J. Ko, T. Parshakova, D. Cai, and R. M. Gower, "SoftServe: A Scalable Quasi-Newton Method for Deep Learning," https://omanscience.com/en/articles/softserve-a-scalable-quasi-newton-method-for-deep-learning.