Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks
This work provides a general unified convergence analysis for GD optimization methods in the training of DNNs with analytic activations such as the softplus and the popular Gaussian error linear unit (GeLU) activation.