A globally convergent regularized Newton method with positive definite regularization for solving nonsmooth optimization problems that replaces the identity matrix in traditional algorithms with a general positive-definite symmetric matrix to regularize the generalized Hessian.
This paper proposes a general line-search Newton framework for unconstrained optimization that avoids repeated Hessian regularization by exploiting the Newton direction only when it is well-defined and suitable and provides the first Newton-type algorithm together with a comprehensive convergence analysis for this important class of nonconvex optimization problems.
This work can specifically ensure, without any smoothness assumptions, convergence to Mordukhovich stationarity as long as the base directions asymptotically revert to the negative gradient for small stepsizes.
This paper proposes a Wolfe-type spectral conjugate gradient method for nonsmooth convex optimization, built on the Moreau-Yosida regularization of the objective function. The method combines a safeguarded spectral parameter with a Dai-Kou-type conjugate parameter, and uses a Wolfe-type line search compatible with the inexact gradients that the regularization produces. We establish global convergence of the method, together with an R-linear convergence rate under an additional strong-convexity assumption. The method is evaluated on standard nonsmooth optimization benchmarks and on large-scale problems, and compared against several existing conjugate gradient and bundle-type methods. The results show that the proposed method performs competitively overall, matching or outperforming existing methods on most problems tested, while a few specific limitations of the current implementation are also identified and discussed
Empirical results reveal that VR-DR remains highly effective for nonsmooth loss functions, significantly broadening its practical utility beyond its theoretical constraints.
Zehui Jia, Denghui Li, Zhiyu Liu et al.· Journal of Scientific Comput...· 0 citations
The quadratic optimization-free (QO-free) method is a class of powerful and effective algorithms for solving nonlinearly constrained optimization problems in Euclidean spaces. The aim of the present work is to extend this method to solve optimization problems on manifolds with additional equality and inequality constraints. We first present a specific algorithm in the manifold setting. At each iteration, three linear systems sharing a common linear operator are solved to determine the master search direction. In addition, a higher-order correction direction is obtained by solving a reduced linear least squares subproblem to circumvent the Maratos effect which is assumed not to arise in existing related literature. A Riemannian arc search is then performed within the tangent space of the current iterate to generate the new iterate. Under appropriate assumptions, we establish the global and strong convergence of the proposed method. Moreover, we prove that the unit step size will eventually be accepted by the arc search, upon which the superlinear convergence of the algorithm is established. Finally, numerical results demonstrate that the proposed method is very competitive compared with other existing approaches.
Chunming Tang, Hao He, Wen Huang et al.· 0 citations
In this paper, we propose a balanced augmented Lagrangian method based on accelerated stochastic ADMM (b-ASADMM) to efficiently solve structured separable nonconvex optimization problems subject to linear constraints. The objective function in this problem comprises potentially nonsmooth and smooth functions, where the smooth function is an average of multiple nonconvex smooth functions. The involved smooth subproblem is tackled by an accelerated stochastic gradient method based on weighting of stochastic item and pre-variable. The involved nonsmooth subproblem is solved under incorporation of Bregman distance to avoid the case that subproblem does not have a closed-form solution due to the complicated quadratic term or other hindering. The involved balanced augmented Lagrangian method advances the original ALM by balancing its subproblems and improving its implementation. In contrast to most deterministic and stochastic ADMMs, our dual variable allows a more flexible and larger step-size region. By standard smoothness assumption, we establish the global convergence and iteration complexity of the generated sequence. Furthermore, we provide a linear convergence rate of b-ASADMM under a local error bound condition and the weakly convex property of the nonsmooth component. Numerical experiments on the graph-guided fused Lasso problem and the smooth clipped absolute deviation penalty problem are conducted to verify the effectiveness of b-ASADMM.