A generalized variant of the MUON optimizer involving an arbitrary number of generalized NS steps with polynomials of possibly arbitrary high degree is proposed and studied and an error analysis for MUON with the generalized NS steps is established that provides convergence rates in terms of the number of gradient step...
This work provides a rigorous error analysis for the momentum SGD optimizer and establishes convergence rates for the momentum optimizer in terms of the size of the learning rate, the size of the mini-batch, and the size of the one-point convexity constant.
A deep learning based approximation algorithm for solutions of stochastic partial differential equations (SPDEs) using a deep neural network for every realization of the driving noise process of a SPDE to approximate the solution process of the SPDE under consideration.
C. Beck, S. Becker, Patrick Cheridito et al.· arXiv.org· 16 citations· ⚡2
This work is the key contribution of this work to establish uniform a priori bounds for Adam and to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.
This work provides a general unified convergence analysis for GD optimization methods in the training of DNNs with analytic activations such as the softplus and the popular Gaussian error linear unit (GeLU) activation.
The key innovative new feature in the proof of the analysis are suitable inverse moment bounds for the second moment process in RMSprop that hold not just for all sufficiently large n but hold for every gradient step $n=1,2,3,...$ with all error constants being explicitly specified.
Steffen Dereich, Arnulf Jentzen· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
