This work provides a rigorous error analysis for the momentum SGD optimizer and establishes convergence rates for the momentum optimizer in terms of the size of the learning rate, the size of the mini-batch, and the size of the one-point convexity constant.
Abstract
Stochastic gradient descent (SGD) optimization schemes are the methods of choice for the optimization of deep neural networks (DNNs) in artificial intelligence (AI) systems. Often not the standard SGD method is used but instead suitable accelerated, adaptive, and/or normalized variants of standard SGD such as Adam, AdamW, and MUON are employed to train large scale AI systems in practically relevant settings. The acceleration (higher order convergence speed) in all these popular optimizers relies on the momentum SGD optimizer. In this work we provide a rigorous error analysis for the momentum SGD optimizer. In particular, we establish convergence rates for the momentum optimizer in terms of the size of the learning rate (step size), the size of the mini-batch, and the size of the one-point convexity constant.
The key innovative new feature in the proof of the analysis are suitable inverse moment bounds for the second moment process in RMSprop that hold not just for all sufficiently large n but hold for every gradient step $n=1,2,3,...$ with all error constants being explicitly specified.
A generalized variant of the MUON optimizer involving an arbitrary number of generalized NS steps with polynomials of possibly arbitrary high degree is proposed and studied and an error analysis for MUON with the generalized NS steps is established that provides convergence rates in terms of the number of gradient step...
This thesis develops a principled grounding for Adam and motivates new designs, and reveals new local structures in matrix-based nonconvex problems, and helps understand and improve recent NN optimizers, such as Muon.
AOS-R (Adaptive Optimizer Switching, Rule-Based), a lightweight controller that monitors six online gradient-space signals and switches among AdamW, SGD-M, and Lion as the optimization landscape evolves, and achieves best accuracy on 6 of 8 combinations with a mean +0.4 pp gain.
A. K. Pandey, Umang Chaturvedi, A. Rana et al.· 0 citations
This work evaluates the subspace Levenberg-Marquardt algorithms for regression and classification tasks in neural networks and compares the performance of subspace LM variants with the classical LM method, as well as other popular first-order algorithms, such as stochastic gradient descent and Adam.
Optimization plays a critical role in training deep neural networks, directly impacting convergence speed, model generalization, and stability. While existing methods such as stochastic gradient descent (SGD) and adaptive optimizers like Adam and AdamW have achieved significant success, they exhibit limitations in hand...
Ahmad Raza Khan, Sarab Almuhaideb· Electronics· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
