The obtained results broaden the applicability of stochastic finite-difference methods to nonsmooth quasiconvex optimization problems and provide a rigorous theoretical justification of the algorithm in black-box settings where only noisy function evaluations are available.
This paper investigates stochastic approximation methods based on finite differences for the minimization of quasiconvex functions. Traditional approaches to convex optimization problems primarily rely on exact or stochastic subgradients. However, in many practical situations, only noisy information about function values is available. In such cases, stochastic quasi-gradient schemes constructed using randomized finite-difference estimators are considered. In particular, we study batch two-point schemes that provide unbiased approximations of the gradient of a smoothed objective function. Variance bounds are established for these estimators, which enable a rigorous convergence analysis of projected stochastic descent methods.
The paper considers the minimization problem of functions of the form $f(x) = \max_{i \in I} f_i(x),$ where each function $f_i(x),$ $i \in I,$ is quasiconvex and has a Lipschitz-continuous gradient on a convex compact set. The main result of the paper is the derivation of convergence rate estimates for stochastic finite-difference methods in the quasiconvex setting. It is shown that the expected suboptimality decreases at the rate $O\!\left(1/\sqrt{k}\right).$ The obtained results significantly broaden the applicability of stochastic finite-difference methods to nonsmooth quasiconvex optimization problems and provide a rigorous theoretical justification of the algorithm in black-box settings where only noisy function evaluations are available.
Rafik A. Khachatryan· Russian Universities Reports...· 0 citations
This paper proposes a novel decentralized stochastic first-order optimization algorithm, which does not require second-order Hessian or Jacobian matrices, for the setting where the lower-level loss function is nonconvex but satisfies the Polyak–Łojasiewicz (PL) condition.
Yihan Zhang, Xinwen Zhang, My T. Thai et al.· 0 citations
We consider the design of optimal fixed-step first-order methods for $M$-Lipschitz convex optimization given $\|x_0-x_\star\|\leq D$. Prior works have identified several distinct fixed-step methods, parameterized by a matrix of stepsizes $W$, with the (information-theoretic) minimax optimal rate $MD/\sqrt{N+1}$ of objective gap convergence. We provide a complete characterization of every optimal fixed-step method. Moreover, we show every optimal fixed-step method can be derived from the constructive approach of~\cite{constructive_approach} and provide a polyhedral representation of the set of optimal methods through proof multipliers. From this characterization, we show that no anytime optimal fixed-step subgradient methods exist.
The conditional gradient method is attractive when linear minimization over the feasible region is substantially cheaper than projection. Its classical convergence theory, however, is formulated for compact feasible sets, whereas many natural convex feasible regions are closed and unbounded. This paper studies a simple compact-restriction principle for applying conditional gradient steps to unbounded feasible regions. The first scheme uses one compact convex set containing the initial objective sublevel set. The second scheme updates the restriction by intersecting compact convex sets generated along the iterations. In both cases the linear minimization oracle is solved only over compact subsets, but the resulting objective values converge to the global optimum of the original problem, provided the compact restrictions contain the corresponding objective sublevel sets. For smooth convex objectives we obtain the standard superlinear convergence rate of the objective. We also record constructive restrictions based on strong convexity, exact sublevel sets, and epigraph caps, and include a nonsmooth conditional subgradient extension with a sublinear convergence rate under a curvature assumption. Numerical experiments illustrate the behaviour of the fixed and dynamic restrictions on unbounded feasible regions.
This work introduces a proximal stochastic subgradient method for minimizing the sum of an expected cost, whose integrand is potentially nonsmooth and nonconvex, and a lower semicontinuous, prox-bounded function. We target a broad class of integrands obeying a nonsmooth, localized variant of the descent lemma in the decision variable, a structural assumption that simultaneously covers smooth losses with Lipschitz gradient and differences of such losses with convex functions. At each iteration the expected cost is replaced by a sample average that is progressively refined, and the proximal-subgradient stepsize is selected by an Armijo-type line search enforcing a sufficient-decrease property up to stochastic errors induced by the sample-based approximation. This framework accommodates substantially more general problem formulations than existing methods, in particular, it requires neither (weak) convexity of the regularizer nor a uniform bound on the variance of the stochastic oracle, and our analysis yields convergence guarantees that are new even in the smooth setting. Specifically, we establish almost sure convergence of the sequence of function values and stationarity of every accumulation point of the trajectories under the relaxed requirement that the sample-size sequence be merely nondecreasing and unbounded, with no prescribed growth rate. Leveraging the Kurdyka-Lojasiewicz (KL) property, we further upgrade this subsequential guarantee to convergence of the whole trajectory to a single stationary point. Finally, for exponential-type KL desingularizing functions and polynomially growing sample sizes, we derive explicit polynomial convergence rates, up to a logarithmic factor, for both the function values and the iterates.
Felipe Atenas, Alejandro Jofré, Pedro Pérez-Aros et al.· 0 citations
In this paper, we study last-iterate convergence of stochastic first-order methods for constrained smooth convex--concave minimax optimization under the standard bounded-variance stochastic oracle. A fundamental challenge is that the last iterates of vanilla stochastic extragradient (S-EG) and stochastic optimistic gradient descent--ascent (S-OGDA) may fail to converge in the presence of stochastic gradient noise, even for simple bilinear problems. To overcome this difficulty, we introduce a simple perturbation framework that regularizes the original convex--concave problem into a strongly convex--strongly concave one. Applying S-EG and S-OGDA to the perturbed problem yields two simple single-loop methods, referred to as perturbed S-EG (PS-EG) and perturbed S-OGDA (PS-OGDA). We establish last-iterate convergence by first deriving convergence in terms of the squared distance to the saddle point of the perturbed problem and then translating this estimate into guarantees for the restricted primal--dual gap. Based on this framework, we establish two types of convergence guarantees. When the optimization horizon is known \emph{a priori}, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/4})$ last-iterate convergence rate for the restricted primal--dual gap, which coincides with the standard primal--dual gap on compact feasible domains. When the optimization horizon is unknown, we develop an anytime variant based on diminishing perturbations and diminishing stepsizes. For general closed convex feasible sets, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/5})$ last-iterate convergence rate for the restricted primal--dual gap. Furthermore, in the unconstrained setting, PS-EG admits a sharper $\mathcal{O}(T^{-1/4})$ anytime convergence rate in terms of the gradient norm.
Taoli Zheng, Jiajin Li, Anthony Man-Cho So· 0 citations
