Stable Perturbation-Robust Policy Optimization (SPrPO) is introduced, which applies adaptive and sensitivity-aware perturbations during RL training and shows improved perturbation robustness while maintaining stable policy optimization.
Abstract
Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.
In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate this issue, behavior policy search has been proposed to learn data-collecting policies tailored to reduce online evaluation variance. However, these approaches do not ac...
Claire Chen, S. Liu, Licheng Luo et al.· 2 citations
Rollout-Time Intervention is introduced, a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary.
Xi Chen, Rui-Xue Ding, Zhigaoyuan Wang et al.· 0 citations
This work first study PG of domain randomized LQR over history-dependent policy classes, such as finite impulse response controllers, as they can extend the possibilities of simultaneous stabilization, and proposes a curriculum learning based algorithm which gradually expands the memory of the controller.
Tesshu Fujinami, Bruce D. Lee, Anastasios Tsiamis et al.· arXiv.org· 0 citations
This work introduces Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts, and establishes minimal intervention as an effective paradigm for enhancing on-policy RL.
Ming-Yu Chen, Ye-Fan Tao, Gerald Friedland et al.· 1 citation
This work formalizes the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and shows that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate.
Christophe D. Hounwanou, John Emeka Eze, Yaé Ulrich Gaba· 1 citation
Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent, is introduced and it is proved that it has the same unique optimal solution as the original PMD objective.
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
MIT News · Artificial Intelligence· news.mit.eduSep 29, 2026
Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.