The proposed AdaBBNC incorporates a flexible BB-based curvature estimate into the proximal gradient framework to enhance adaptability in nonconvex settings and achieves the optimal iteration complexity of $\mathcal{O}(\epsilon^{-2})$ for finding an $\epsilon$-stationary point, without requiring any prior knowledge of the global Lipschitz constant.
Abstract
The Barzilai-Borwein (BB) method is an efficient gradient-based approach for unconstrained optimization that approximates spectral information of the Hessian matrix to capture curvature at low computational cost. In this paper, we extend the BB stepsize strategy to composite nonconvex optimization problems consisting of a smooth nonconvex term and a proper closed convex term, and propose an adaptive Barzilai-Borwein proximal gradient method for nonconvex optimization (AdaBBNC). The proposed method incorporates a flexible BB-based curvature estimate into the proximal gradient framework to enhance adaptability in nonconvex settings. Under mild assumptions, we establish that AdaBBNC achieves the optimal iteration complexity of $\mathcal{O}(\epsilon^{-2})$ for finding an $\epsilon$-stationary point, without requiring any prior knowledge of the global Lipschitz constant. Numerical experiments demonstrate the effectiveness and robustness of the proposed method. Compared with recent parameter-free and line-search-free adaptive proximal gradient methods, AdaBBNC exhibits more aggressive yet stable behavior in ill-conditioned optimization problems.
Bregman proximal stochastic gradient (BPSG) methods bring variance-reduced composite optimization to objectives whose geometry is poorly captured by Euclidean smoothness. Their performance, however, remains sensitive to the step size: raw stochastic curvature estimates can fluctuate sharply, whereas line searches add repeated proximal evaluations. We introduce Ada-BPSG, a line-search-free BPSG method that couples the SAGA gradient table with a stabilized Barzilai--Borwein (BB) candidate. A mediant aggregates incremental secant information so that nearly singular local ratios receive little weight, and an explicit safeguard translates the resulting curvature estimate into the bounded step-size sequence required for convergence. This design yields a direct analytical chain from relative smoothness and component-wise variance control to convergence in finite-dimensional normed spaces. We prove an $O(n/K)$ ergodic rate for convex objectives, a restarted linear rate under relative quadratic growth, and an $O(1/K)$ bound for a Bregman proximal residual in the nonconvex setting. On logistic regression and sparse nonnegative matrix factorization, Ada-BPSG combines low objective values with substantially less sensitivity to the initial step size than standard variance-reduced baselines, while avoiding line search.
Chenhan Jin, Shengze Xu, Binghui Xie et al.· 0 citations
The paper proposes a novel proximal difference-of-convex (DC) algorithmic framework to solve general non-convex, non-smooth optimization problems. By combining Barzilai-Borwein (BB) step sizes with nonmonotone line search strategies, our approach effectively overcomes the conservative step sizes and stability issues inherent in standard proximal DC algorithms. Furthermore, we develop extrapolation mechanisms to accelerate convergence while ensuring global stability. The global convergence of the proposed algorithms is rigorously established under the Kurdyka-\L ojasiewicz property. Numerical experiments on the SCAD-regularized least squares problem and graphic Ginzburg-Landau image segmentation models demonstrate that the proposed methods achieve highly competitive efficiency and accuracy compared to existing DC algorithms.
This work introduces a proximal stochastic subgradient method for minimizing the sum of an expected cost, whose integrand is potentially nonsmooth and nonconvex, and a lower semicontinuous, prox-bounded function. We target a broad class of integrands obeying a nonsmooth, localized variant of the descent lemma in the decision variable, a structural assumption that simultaneously covers smooth losses with Lipschitz gradient and differences of such losses with convex functions. At each iteration the expected cost is replaced by a sample average that is progressively refined, and the proximal-subgradient stepsize is selected by an Armijo-type line search enforcing a sufficient-decrease property up to stochastic errors induced by the sample-based approximation. This framework accommodates substantially more general problem formulations than existing methods, in particular, it requires neither (weak) convexity of the regularizer nor a uniform bound on the variance of the stochastic oracle, and our analysis yields convergence guarantees that are new even in the smooth setting. Specifically, we establish almost sure convergence of the sequence of function values and stationarity of every accumulation point of the trajectories under the relaxed requirement that the sample-size sequence be merely nondecreasing and unbounded, with no prescribed growth rate. Leveraging the Kurdyka-Lojasiewicz (KL) property, we further upgrade this subsequential guarantee to convergence of the whole trajectory to a single stationary point. Finally, for exponential-type KL desingularizing functions and polynomially growing sample sizes, we derive explicit polynomial convergence rates, up to a logarithmic factor, for both the function values and the iterates.
Felipe Atenas, Alejandro Jofré, Pedro Pérez-Aros et al.· 0 citations
A globally convergent regularized Newton method with positive definite regularization for solving nonsmooth optimization problems that replaces the identity matrix in traditional algorithms with a general positive-definite symmetric matrix to regularize the generalized Hessian.
This paper proposes a general line-search Newton framework for unconstrained optimization that avoids repeated Hessian regularization by exploiting the Newton direction only when it is well-defined and suitable and provides the first Newton-type algorithm together with a comprehensive convergence analysis for this important class of nonconvex optimization problems.
We study Langevin-based methods for non-convex optimization under smoothness and dissipativity assumptions. Our focus is on obtaining non-asymptotic bounds for the expected excess risk rather than sampling guarantees for the full target distribution. The key ingredient of our analysis is a direct passage from relative entropy to objective-value error, based on a weighted Csisz\'ar--Kullback--Pinsker inequality and exponential-moment estimates. This avoids intermediate Wasserstein bounds and yields sharper dependence on the Log-Sobolev constant, a quantity that may scale exponentially with the inverse temperature and the dimension in non-convex problems. We first analyze the Unadjusted Langevin Algorithm with exact gradients and derive explicit bounds on $\mathbb{E}[F(x_k)]-\min F$ in terms of the inverse temperature, dimension, stepsize, smoothness and dissipativity parameters, and the Log-Sobolev constant. We then extend the result to an inexact-gradient version of ULA, allowing for biased and stochastic gradient surrogates whose mean-square error grows at most quadratically in the state. This framework covers stochastic gradients and zeroth-order estimators based only on function evaluations. In particular, we show that both Gaussian and spherical finite-difference estimators fit into the inexact-ULA theory and obtain explicit function-evaluation complexity bounds for zeroth-order Langevin optimization. To the best of our knowledge, these are the first non-asymptotic global non-convex optimization complexity bounds for zeroth-order ULA. We also provide numerical experiments illustrating the behavior of the proposed zeroth-order Langevin schemes.
E. Naldi, Marco Rando, Lorenzo Rosasco et al.· 0 citations