This appears to be the first time that a stochastic model-free policy optimization method for LQR converges with high probability with $\tilde{O}(1)$ per-iteration computation and polylogarithmic dependence on the confidence level.
Abstract
We study the convergence of the vanilla stochastic policy gradient method applied to the linear quadratic regulator (LQR) problem. The method is cheap in the following sense: (1) at each iteration only $\tilde{O}(1)$ interactions with the environment are needed, therefore allowing frequent policy improvement steps, and (2) to ensure stability throughout and convergence to an $\epsilon$-optimal policy with probability $1-\delta$, only $O(\mathtt{Polylog}(1/\delta)/\epsilon)$ interactions are needed. To the best of our knowledge, this appears to be the first time that a stochastic model-free policy optimization method for LQR converges with high probability with $\tilde{O}(1)$ per-iteration computation and polylogarithmic dependence on the confidence level. The convergence analysis presented here is agnostic to LQR specifics and hence could be potentially generalized to a broader class of problems.
We study the time-uniform convergence of the raw iterate of standard stochastic gradient descent (SGD) for unconstrained smooth convex objectives. We prove that, under standard noise assumptions, the time-uniform convergence rate gets arbitrarily close to $\sqrt{\log n / n}$ but never reaches it. More specifically, we...
Success conditioning is a strategy for improving decision-making policies in stochastic environments; it updates a policy by increasing the probability of taking actions that yield successful outcomes. Success conditioning is common to many reinforcement learning applications, yet its limiting behavior and convergence...
We study second-order methods for convex stochastic optimization, where gradients and Hessians are available only through stochastic estimates with variances $\sigma_1^2$ and $\sigma_2^2$, respectively. First, we propose the Stochastic Contracting Cubic Newton method. At each iteration, it minimizes a cubic model with...
A. Agafonov, Аслан Нажмудинович Ашабоков, D. Kamzolov et al.· 0 citations
We analyze a stochastic algorithm with Halpern-type anchoring for constrained convex-concave problems and monotone variational inequalities. This single-loop and single-call algorithm uses one unbiased sample of the gradient operator at every iteration, to be applicable to monotone games with noisy feedback. With $t$ d...
A time-varying quadratic norm is constructed that turns this window contraction into an exact one-step Lyapunov identity, leading to coupled one-step recursions for the centroid and disagreement errors, without unrolling the dynamics over communication windows.
This work studies the stochastic first-order oracle complexity for constrained or regularized convex-concave min-max optimization and stochastic monotone variational inequalities and extends to prove the same complexity for problems without the bounded variance, by using the Blum-Gladyshev assumption.
Ahmet Alacaoglu· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
