The optimal accuracy dependence for smooth nonconvex--strongly-convex bilevel optimization with stochastic first-order oracles is characterized and an $\Omega(\epsilon^{-6})$ lower bound matching the accuracy exponent of existing upper bounds is proved.
Abstract
We characterize the optimal accuracy dependence for smooth nonconvex--strongly-convex bilevel optimization with stochastic first-order oracles. For globally unbiased fresh-gradient observations with bounded variance, we prove an $\Omega(\epsilon^{-6})$ lower bound matching the accuracy exponent of existing upper bounds. We consider globally Lipschitz gradients, a bounded upper gradient in the lower variable, and Lipschitz lower Hessian blocks, with target $\mathbb{E}|\nabla F(\widehat{x})|\le\epsilon$. Let $L$ be the common first-order scale, $\mu$ the lower strong-convexity modulus, $\rho$ the lower Hessian-variation budget, $\kappa=L/\mu$, and $\chi=1+\rho/\mu$. For $\rho\gtrsim\mu$, $\chi\lesssim\kappa$, sufficiently high accuracy, and sufficiently large dimension, we establish $\Omega\left(L\kappa^2\Delta\epsilon^{-2}+L^3\chi^2\kappa^8\sigma_g^2\Delta\epsilon^{-6}\right)$, where $\Delta$ is the initial gap budget and $\sigma_g^2$ is the lower-gradient variance budget. The bound holds over full Euclidean spaces against arbitrary randomized adaptive algorithms, even when every sample is the gradient of a scalar function. In the common-scale regime $\rho=\Theta(L)$, the leading stochastic term has condition-number dependence $\kappa^{10}$, compared with the achievable $\kappa^{11}$ dependence. The proof embeds a sequential hard objective into an exact lower response while preserving global regularity and finite gap. A multiscale decomposition limits the gradient signal carrying each new direction, yielding the required sample complexity. We complement this lower bound with a frozen-linear-tilt estimator whose bias is proportional to lower Hessian variation. Its analysis makes this structural dependence explicit and yields matching leading rates in the specified curvature-controlled and small-gap regimes.
The bounded-variance lower bound improves the dependence on the condition number from $\kappa^{1/3}$ in previous lower bounds to $\kappa$, while the averaged-smoothness lower bound is the first of its kind.
We study the sample complexity of finding a point with expected hypergradient norm at most $\epsilon>0$ in nonconvex--strongly-convex stochastic bilevel optimization using fresh, globally unbiased first-order samples with bounded variance. For lower-variable smoothness order $p\ge1$, F$^2$SA-$p$ achieves $\widetilde O(...
Wen-Dao Wu, Hai-Han Zhang, Chen-Heng Zhang et al.· 0 citations
We characterize the fresh-gradient oracle complexity of smooth nonconvex-strongly-concave minimax optimization, with matching upper and lower bounds up to logarithmic factors. Let $\Phi(x)=\max_y f(x,y)$, where $f$ is jointly $L$-smooth and $\mu$-strongly concave in $y$ on unconstrained Euclidean domains, and set $\kap...
Wen-Dao Wu, Hai-Han Zhang, Chen-Heng Zhang et al.· 2 citations
We study whether the linear condition-number dependence in the stochastic complexity of SAPD+ is necessary for nonconvex-strongly-concave minimax optimization. For jointly $L$-smooth objectives with dual strong-concavity parameter $\mu$, we prove a lower bound that matches the SAPD+ upper bound under the same Moreau-en...
We establish a randomized first-order lower bound for finding an $\epsilon$-stationary point, $\|\nabla f(x)\|\le \epsilon$, of a nonconvex function with initial gap at most $\Delta$, $L_1$-Lipschitz gradient, and $L_2$-Lipschitz Hessian. Each oracle call returns the exact function value and gradient. Let $Q_{\mathrm{r...
We characterize the oracle complexity of finding first-order stationary points in smooth nonconvex--strongly-concave (NC--SC) minimax optimization when each query provides the full Hessian in addition to the function value and gradient. For $\Phi(x)=\max_y f(x,y)$, assume that $f$ has an $L$-Lipschitz gradient, a $\rho...
Fang-Qing Zhang, Hai-Han Zhang, Wen-Dao Wu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.