Hyper-Gradient Methods for Bilevel Optimization with Manifold Lower-level Solution Set

Minima Selection in Stochastic Optimization: A Long-Time, Small-Stepsize Perspective

Efficient Hessian-Free Methods for Multi-Objective Bilevel Optimization with Nonconvex Lower Level

This work proposes a class of Multi-Objective Moreau Envelope based Hessian-free Algorithms (MOMEHA) to solve the multi-objective bilevel learning problems with nonconvex lower level and proposes a momentum-based variant of MOMEHA (i.e., MB-MOMEHA) method to solve the stochastic multi-objective bilevel learning problems.

Convergence Analysis of Decentralized Hessian-/Jacobian-Free Algorithm for Nonconvex Stochastic Bi-Level Optimization

This paper proposes a novel decentralized stochastic first-order optimization algorithm, which does not require second-order Hessian or Jacobian matrices, for the setting where the lower-level loss function is nonconvex but satisfies the Polyak–Łojasiewicz (PL) condition.

A Minimal-Gradient Subspace Method for Unconstrained Optimization

A minimal-gradient subspace method for unconstrained optimization of SPD quadratics, which attains the highest success count, whereas L-BFGS requires fewer median gradient evaluations and less CPU time.

Stochastic Dynamic Barrier Perturbed Gradient Methods for Nonconvex Simple Bilevel Optimization

PR-SDBPG, a penalty-regularized variant that eliminates the rare-visit assumption, and VR-PR-SDBPG, which improves the resulting sample complexities entirely through variance reduction, are developed, believed to be the first explicit stochastic nonconvex-nonconvex simple bilevel optimization guarantees.

First-Order Methods for Distributionally Robust Constrained Optimization

This paper proposes a tractable stochastic approach based on an entropic regularization of the distributionally robust value function, which makes it possible to compute stochastic gradient estimators, and the combination of these estimators with a stochastic Frank-Wolfe algorithm, allowing us to optimize the regularized robust objective while naturally handling constraints.