This paper proposes a general line-search Newton framework for unconstrained optimization that avoids repeated Hessian regularization by exploiting the Newton direction only when it is well-defined and suitable and provides the first Newton-type algorithm together with a comprehensive convergence analysis for this important class of nonconvex optimization problems.
Abstract
Newton's method is one of the most effective second-order algorithms for smooth optimization because of its fast local convergence. However, existing globally convergent Newton-type methods typically require convexity or strong convexity of the objective function, while approaches for nonconvex optimization often rely on Hessian regularization at every iteration. In this paper, we propose a general line-search Newton framework for unconstrained optimization that avoids repeated Hessian regularization by exploiting the Newton direction only when it is well-defined and suitable. The proposed framework encompasses several existing hybrid gradient--Newton methods as special cases and naturally yields a new extragradient Newton method. We establish global convergence under mild assumptions, including the Polyak--Lojasiewicz--Kurdyka (PLK) condition, allowing both isolated and nonisolated accumulation points. We further prove local superlinear and quadratic convergence under appropriate regularity assumptions. Finally, we apply the proposed framework to strongly quasiconvex optimization and provide, to the best of our knowledge, the first Newton-type algorithm together with a comprehensive convergence analysis for this important class of nonconvex optimization problems. Numerical experiments demonstrate the effectiveness of the proposed methods.
We develop a new direct accelerated Newton method for minimizing convex functions with Lipschitz continuous Hessian. The algorithm uses only primal variables and performs just one linear solve per iteration. With a simple predetermined choice of parameters, it achieves the global convergence rate of $O(1/k^3)$ in terms of the functional residual. To the best of our knowledge, this is the first second-order method for this problem class attaining this rate while relying solely on one linear system solve per iteration (without solving auxiliary nonlinear regularized subproblems, such as cubic regularization, performing nonlinear parameter searches, or using dual extragradient corrections). Our method can be implemented in a Hessian-free way, using an inexact linear system solver, while preserving the fast global rate. We further extend our construction to arbitrary geometry through Bregman divergence, and to composite optimization problems.
For solving nonconvex equality-constrained optimization problems, a recent Gradient-Eigenstep Algorithm by Goyens et al.~is an iteration-efficient approach, based on minimizing Fletcher's augmented Lagrangian function, for finding an approximate second-order stationary point from an arbitrary starting point. In this paper, the analysis of this algorithm is extended, offering a two-fold contribution. First, it is shown that a local-linear rate of convergence can be obtained by this method if it is initiated sufficiently close to a strong second-order stationary point and employs a sufficiently small step-size parameter and sufficiently large penalty parameter. In this case, the algorithm reduces to a gradient descent algorithm applied to minimize Fletcher's augmented Lagrangian. Second, as a particularly useful application of the first result, it is shown that the Gradient-Eigenstep algorithm can be used as an iteration-efficient subproblem solver in the context of a progressive sampling strategy for solving equality-constrained optimization problems when the objective and constraint functions are defined by large sample averages, ultimately offering an algorithm with an improved worst-case sample complexity when compared to an approach that solves a full-sample problem directly.
Frank E. Curtis, Ling-Jun Guo, Daniel P. Robinson· 0 citations
A globally convergent regularized Newton method with positive definite regularization for solving nonsmooth optimization problems that replaces the identity matrix in traditional algorithms with a general positive-definite symmetric matrix to regularize the generalized Hessian.
This work can specifically ensure, without any smoothness assumptions, convergence to Mordukhovich stationarity as long as the base directions asymptotically revert to the negative gradient for small stepsizes.
This work proposes a nonlinear-residual linearized augmented Lagrangian method (NR-LALM) that replaces this subproblem by a regularized Gauss-Newton-type step while retaining the classical multiplier update based on the nonlinear constraint residual.
Benqi Liu, Kangkang Deng, Zichen Wang et al.· 0 citations
The main contribution is a finite-time mechanism for converting stationarity of the truncated minimax problem into a KKT certificate for the original constrained problem and establishing explicit convergence rates for the proposed method in terms of the KKT residual.