Skip to content

SoftServe: A Scalable Quasi-Newton Method for Deep Learning

Oct 2026 · 0 citations · 49 references
Computer Science

TL;DR

SoftServe, a family of QN methods designed to overcome non-convexity and enormous parameter sizes, is introduced, and derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature.

Abstract

Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.

View source

Similar papers

#machine learning Preprint Oct 2026

Scalable Levenberg-Marquardt Training Framework for Deep Learning

The Levenberg-Marquardt (LMA) algorithm is a widely used nonlinear least-squares solver known for its rapid convergence. For small-scale neural networks, LMA provides better convergence and architectural robustness than first-order optimizers like Adam. However, LMA imposes prohibitive memory and computational costs. T...

Xiang-Hao Wang, Johan Barthélemy, John A. Taylor · 0 citations
#machine learning Preprint Sep 2026

Learned Preconditioning for a Primal-Dual Interior-Point Method

Interior-point methods (IPMs) are among the most widely used algorithms for constrained optimization, yet their Newton-based search directions require costly second-order information and large linear-system solves. Learning to optimize offers cheaper updates learned from data, but the singular behavior of logarithmic b...

A. Madabhushi, Jia-Lin Liu, Min-Xin Zhang · 0 citations
#artificial intelligence Preprint Sep 2026

Matrix AdaGrad: Row-wise and Column-wise Adaptive Subgradient Methods

An Online Mirror Descent framework with adaptive proximal functions for matrix-valued parameters, providing a principled methodology for deriving matrix-aware adaptive optimization through online regret minimization and yields Row-wise Matrix AdaGrad and Column-wise Matrix AdaGrad as concrete instantiations with regret...

Wen-Peng Zhang, Run-Sheng Yu, Pei-Lin Zhao · 0 citations
#artificial intelligence Preprint Sep 2026

Beyond the Matrix Sign: Quadratic Spectral Descent

Muon emerges as a strong competitor of the AdamW for LLM pretraining, because the matrix-wise update it employs can potentially incur smaller second-order penalty than the once dominating AdamW, which performs coordinate-wise update. However, the spectral flattening procedure in Muon is quite debatable since it discard...

Qiao-Zhe Zhang, Jun Sun, Ying-Zhuang Liu · 2 citations
Preprint Sep 2026

DREAM: Deep-Reparametrization of Adaptive Regularization Maps for Fast Zero-Shot Self-Supervised Learning

Adaptive regularization is an effective means of improving the flexibility of classical variational reconstruction methods while retaining their interpretability and mathematical structure. In this work, we propose an approach for learning deep-reparameterized adaptive regularization maps (DREAM) for Total Variation (T...

T. Vu, A. Biguri, Christoph Kolbitsch et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.