Jul 2026· Applied and Computational Engineering· 0 citations
TL;DR
This paper deeply integrates convex optimization theory with the backpropagation algorithm and constructs a novel stable and efficient training mechanism for neural networks that achieves favorable adaptability to both shallow fully connected networks and deep convolutional networks.
Abstract
Traditional neural network training based on backpropagation suffers from multiple bottlenecks, including slow convergence rate, susceptibility to local optima, vanishing/exploding gradients, and insufficient generalization performance. To address these issues, this paper deeply integrates convex optimization theory with the backpropagation algorithm and constructs a novel stable and efficient training mechanism for neural networks. Systematical optimization of the conventional training pipeline is realized via convex reconstruction of the loss function, design of an adaptive gradient correction rule under convex optimization constraints, and rigorous theoretical proof of convergence for the integrated algorithm. Experimental results demonstrate that compared with mainstream algorithms such as standard BP, SGD and Adam, the proposed mechanism reduces the number of convergence iterations by over 35%, cuts training time by 28%, improves classification accuracy by 4%-7%, and effectively suppresses gradient anomalies. It achieves favorable adaptability to both shallow fully connected networks and deep convolutional networks. This research complements the theoretical convex optimization framework for non-convex training, and provides methodological support and theoretical references for efficient training and industrial deployment of deep learning models.
Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence. This paper shows that indeed the simplest Monte Carlo algorithm implemented on a single GPU -- randomly mutate a parameter, keep it if the loss decreases, otherwise retry -- can practically train deep networks. This gradient-free method does not even need common techniques such as batch normalization or residual connections to directly train sufficiently deep networks. More remarkably, its flexibility extends to several nontrivial scenarios: it enables pure pruning training, supports discrete weights, accommodates unconventional transfer functions such as Gaussian, and reveals the substantial redundancy of deep networks. We have demonstrated its feasibility on deep networks with more than 20 layers, single-hidden-layer wide networks with up to 16,384 hidden neurons, and even a simple Transformer architecture trained on both image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This simple gradient-free method may offer a complementary perspective for understanding the self-organization and learning mechanisms of neural networks, and also provides an alternative route for building physically inspired deep learning systems.
Deep neural networks often encounter non-convex optimization challenges during training due to the presence of local minima, saddle points, and complex loss surfaces. Existing optimization algorithms such as Adam and Stochastic Gradient Descent (SGD) offer complementary advantages—Adam provides faster convergence, while SGD tends to achieve better generalization. However, neither optimizer alone effectively balances both properties in non-convex settings. To address this limitation, this paper proposes a phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD. The proposed method employs Adam during the initial phase of training to enable rapid convergence and efficient exploration of the loss landscape, and then switches to momentum-based SGD in the later phase to improve generalization and ensure stable convergence. The effectiveness of the proposed approach is evaluated on one benchmark dataset, MNIST dataset, under different learning rate settings. Experimental results demonstrate that the proposed method achieves performance that is superior or comparable to existing optimizers in terms of accuracy and loss minimization. These results indicate that the proposed hybrid optimization strategy provides a simple and effective solution for handling non-convex optimization problems in deep learning.
Deep neural networks often encounter non-convex optimization challenges during training due to the presence of local minima, saddle points, and complex loss surfaces. Existing optimization algorithms such as Adam and Stochastic Gradient Descent (SGD) offer complementary advantages—Adam provides faster convergence, while SGD tends to achieve better generalization. However, neither optimizer alone effectively balances both properties in non-convex settings. To address this limitation, this paper proposes a phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD. The proposed method employs Adam during the initial phase of training to enable rapid convergence and efficient exploration of the loss landscape, and then switches to momentum-based SGD in the later phase to improve generalization and ensure stable convergence. The effectiveness of the proposed approach is evaluated on three benchmark dataset the M4 time-series forecasting data set, under different learning rate settings. Experimental results demonstrate that the proposed method achieves performance that is superior or comparable to existing optimizers in terms of accuracy and loss minimization. These results indicate that the proposed hybrid optimization strategy provides a simple and effective solution for handling non-convex optimization problems
Findings suggest that proximity minimisation may provide a useful basis for stabilising optimisation when fixed-point assumptions are difficult to satisfy, but stronger mathematical proofs, complete ablation studies, consistent experimental specifications, and independently reproducible implementations are required to establish the method’s broader reliability and scalability.
S. Nirmalkumar· Asian journal of mathematics...· 0 citations
Recent advancements in machine learning and deep neural networks have increased the need for efficient optimization techniques, particularly adaptive learning rate methods. The learning rate plays a critical role in determining convergence speed, training stability, computational efficiency, and model generalization. Traditional fixed learning rate approaches often experience slow convergence and instability in deep learning applications. To overcome these limitations, adaptive optimization algorithms such as AdaGrad, RMSProp, AdaDelta, Adam, Nadam, and AMSGrad were introduced. These methods dynamically adjust learning rates using gradient statistics and momentum mechanisms, enabling faster and more stable optimization in complex and high-dimensional learning environments. This paper reviews adaptive learning rate methods developed before 2019, focusing on their mathematical foundations, convergence behavior, computational efficiency, and generalization performance. It compares classical and modern optimization techniques across supervised, unsupervised, reinforcement, and deep learning models. The study also examines their role in handling vanishing and exploding gradients, reducing overfitting, and improving scalability. The findings show that adaptive optimization methods significantly enhance training efficiency compared to conventional gradient descent methods, especially in large-scale deep learning systems. However, some methods may achieve faster convergence at the cost of weaker generalization performance. The paper concludes that adaptive learning rate strategies are essential for modern machine learning applications such as computer vision, NLP, robotics, healthcare analytics, and intelligent automation, while future research should focus on hybrid and meta-learning-based optimization approaches.
Arvind Kumar Singh, Lakshmi Narayanan· International Journal of Int...· 0 citations
Deep neural networks often contain substantial parameter redundancy, resulting in unnecessary computational cost and energy consumption. This work presents a dynamically adaptive gating mechanism for learning layer-wise sparsity through differentiable masking. In the proposed framework, each network parameter is associated with a self-learning gate that controls its contribution during training. The gating function follows a progressive soft-to-hard transition in which the slope is gradually annealed, enabling the model to move smoothly from continuous parameter weighting to near-binary pruning decisions while simultaneously learning an adaptive threshold parameter. The framework is evaluated across multilayer perceptrons (MLPs), deep neural networks (DNNs), Tabular Transformer models, and benchmarked against $\mathrm{L}_{0}$ regularization and Variational Dropout on convolutional neural networks. Experimental results demonstrate substantial model compression without degrading predictive performance, and in some cases improving it. On MNIST, the gated MLP retains only 14% of weights (86% pruning) while maintaining 98% accuracy. The gated DNN and Tab Transformer similarly outperform their dense counterparts while retaining only 46% and 46.22% of parameters, respectively. Benchmarking further shows that the proposed adaptive gating achieves competitive or superior accuracy-sparsity trade-offs compared with $\mathbf{L}_{0}$ regularization and Variational Dropout. These results demonstrate that the proposed dynamically adaptive gating framework provides an efficient and interpretable pathway for sparsity learning, enabling high-performing lightweight neural network deployment.
Raunak Dev, Mydhily Sankar, Devaprabha Biju S et al.· 2026 International Conferenc...· 0 citations