Jul 2026· Asian journal of mathematics and computer research· Vol 33, pp. 318-337· 0 citations
TL;DR
Findings suggest that proximity minimisation may provide a useful basis for stabilising optimisation when fixed-point assumptions are difficult to satisfy, but stronger mathematical proofs, complete ablation studies, consistent experimental specifications, and independently reproducible implementations are required to establish the method’s broader reliability and scalability.
Abstract
Best Proximity Learning (BPL) is presented as an optimisation framework for training deep neural networks under non-convex and constrained conditions. The framework reformulates parameter updating as the minimisation of a proximity gap between the current parameters and their mapped counterparts, rather than requiring an exact fixed point. On this basis, the study develops the Best Proximity Gradient Descent (BPGD) algorithm, which combines stochastic gradient updates, proximity projection, adaptive thresholding, and constraint-aware mappings. Conceptual convergence principles are formulated for expected proximal contraction, cyclic mappings, and adaptive learning rates. The method is evaluated on image classification, adversarial training, federated learning with non-identically distributed client data, and language modelling. The reported experiments use CIFAR-10, CIFAR-100, MNIST, and WikiText-103 with established neural architectures and compare BPGD with SGD, Adam, AMSGrad, and proximal gradient descent. Across the reported tasks, BPGD demonstrates comparatively stable loss trajectories and favourable accuracy, robustness, and perplexity values, while maintaining a per-epoch computational cost close to that of SGD. The proximity-based formulation is also discussed in relation to recurrent networks, convolutional models, transformers, and constrained parameter spaces. These findings suggest that proximity minimisation may provide a useful basis for stabilising optimisation when fixed-point assumptions are difficult to satisfy. However, stronger mathematical proofs, complete ablation studies, consistent experimental specifications, and independently reproducible implementations are required to establish the method’s broader reliability and scalability.
This study presents a novel Deep Proximal Gradient Descent framework for ill-posed problems by employing a tailored second-order differentiable Input-Convex Neural Networks (ICNNs) as a learned regularizer. A key contribution is the design of convex residual mapping, which preserves the convexity of the regularized objective, thereby enhancing the interpretability of the deep network without sacrificing its expressive power. Based on this framework, we develop two types of algorithms. For linear problems, the ICNN-based regularizer is embedded into the standard proximal gradient structure. For nonlinear problems, we introduce an innovative formulation that employs the learned residual to guide gradient descent, while using the traditional data misfit as a proximal regularizer to avoid network-dominated spurious solutions. Building on this iterative scheme, we establish groundbreaking convergence results for both algorithms, complete with rigorous proofs. Extensive numerical experiments, particularly on real low-dose Computed Tomography data, validate the superior imaging quality and high computational efficiency of our algorithms.
T. Ye, Guangyu Gao, Yang Li et al.· Inverse Problems· 0 citations
This paper deeply integrates convex optimization theory with the backpropagation algorithm and constructs a novel stable and efficient training mechanism for neural networks that achieves favorable adaptability to both shallow fully connected networks and deep convolutional networks.
Weiwei Guo· Applied and Computational En...· 0 citations
Deep neural networks often encounter non-convex optimization challenges during training due to the presence of local minima, saddle points, and complex loss surfaces. Existing optimization algorithms such as Adam and Stochastic Gradient Descent (SGD) offer complementary advantages—Adam provides faster convergence, while SGD tends to achieve better generalization. However, neither optimizer alone effectively balances both properties in non-convex settings. To address this limitation, this paper proposes a phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD. The proposed method employs Adam during the initial phase of training to enable rapid convergence and efficient exploration of the loss landscape, and then switches to momentum-based SGD in the later phase to improve generalization and ensure stable convergence. The effectiveness of the proposed approach is evaluated on one benchmark dataset, MNIST dataset, under different learning rate settings. Experimental results demonstrate that the proposed method achieves performance that is superior or comparable to existing optimizers in terms of accuracy and loss minimization. These results indicate that the proposed hybrid optimization strategy provides a simple and effective solution for handling non-convex optimization problems in deep learning.
Generalized smoothness, such as (L0, L1)-smoothness, have recently attracted considerable attention due to their ability to model optimization problems arising in modern machine and deep learning, where the classical Lipschitz assumptions of the gradient is often violated. At the same time, computing exact gradients may be impractical or computationally expensive in many applications. In this work, we study convex (L0, L1)-smooth optimization (for normalized gradient method we consider quasi-convex problems too) under access only to a normalized approximation recently proposed Comparison Oracle, which returns an inexact normalized gradient in linear time with a bounded absolute error. Within this framework, we develop comparison-oracle variants of Normalized Gradient Descent and Gradient Descent with Polyak stepsizes. We establish explicit upper bounds on the approximation error that guarantee convergence and derive convergence rates for all proposed methods. Unlike existing analyses, our results require neither classical smoothness assumptions nor access to exact gradients or their exact normalized counterparts. Finally, numerical experiments corroborate the theoretical findings.
Deep neural networks often encounter non-convex optimization challenges during training due to the presence of local minima, saddle points, and complex loss surfaces. Existing optimization algorithms such as Adam and Stochastic Gradient Descent (SGD) offer complementary advantages—Adam provides faster convergence, while SGD tends to achieve better generalization. However, neither optimizer alone effectively balances both properties in non-convex settings. To address this limitation, this paper proposes a phase-switch hybrid optimization strategy that combines the strengths of Adam and SGD. The proposed method employs Adam during the initial phase of training to enable rapid convergence and efficient exploration of the loss landscape, and then switches to momentum-based SGD in the later phase to improve generalization and ensure stable convergence. The effectiveness of the proposed approach is evaluated on three benchmark dataset the M4 time-series forecasting data set, under different learning rate settings. Experimental results demonstrate that the proposed method achieves performance that is superior or comparable to existing optimizers in terms of accuracy and loss minimization. These results indicate that the proposed hybrid optimization strategy provides a simple and effective solution for handling non-convex optimization problems
Gradient-based learning under non-convex constraints exhibits a notable phenomenon: Despite the existence of many equivalent global minimizers, optimization algorithms consistently converge to a small subset of structured solutions. This behavior, known as implicit bias, remains insufficiently understood in constrained and non-convex settings. In this article, we investigate the mechanism of implicit bias induced by projected gradient-based optimization over general non-convex feasible sets. By modeling projected gradient descent as a continuous-time dynamical system, we derive a projected gradient flow characterized by tangent and normal cone decompositions, which capture the local geometry of the constraint set. Based on this formulation, we show that constraint geometry continuously filters gradient directions along the optimization trajectory, leading to a trajectory-dependent implicit regularization effect without modifying the objective function. We further formalize this effect through a cumulative normal projection energy functional and prove that the optimization dynamics converge to solutions minimizing both empirical risk and geometric incompatibility with the constraint set. Extensive experiments on synthetic and real-world datasets validate the theoretical predictions, demonstrating consistent alignment between solution geometry, optimization trajectories, and generalization performance. These results provide a unified geometric and dynamical explanation of implicit bias in constrained learning systems.