Skip to content

Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

Sep 2026 · 0 citations · 45 references
Computer Science

TL;DR

Stable Perturbation-Robust Policy Optimization (SPrPO) is introduced, which applies adaptive and sensitivity-aware perturbations during RL training and shows improved perturbation robustness while maintaining stable policy optimization.

Abstract

Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.

View source

Similar papers

Preprint Aug 2026

Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation

In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate this issue, behavior policy search has been proposed to learn data-collecting policies tailored to reduce online evaluation variance. However, these approaches do not ac...

Claire Chen, S. Liu, Licheng Luo et al. · 2 citations
#artificial intelligence Preprint Oct 2026

Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

Rollout-Time Intervention is introduced, a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary.

Xi Chen, Rui-Xue Ding, Zhigaoyuan Wang et al. · 0 citations
Sep 2026

Policy Gradient over History-Dependent Policy Classes for LQR with Domain Randomization

This work first study PG of domain randomized LQR over history-dependent policy classes, such as finite impulse response controllers, as they can extend the possibilities of simultaneous stabilization, and proposes a curriculum learning based algorithm which gradually expands the memory of the controller.

Tesshu Fujinami, Bruce D. Lee, Anastasios Tsiamis et al. · 0 citations
#artificial intelligence Review Sep 2026

MInTRL: Off-policy Intervention can boost On-policy RL

This work introduces Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts, and establishes minimal intervention as an effective paradigm for enhancing on-policy RL.

Ming-Yu Chen, Ye-Fan Tao, Gerald Friedland et al. · 1 citation
#artificial intelligence Preprint Aug 2026

Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

This work formalizes the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and shows that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate.

Christophe D. Hounwanou, John Emeka Eze, Yaé Ulrich Gaba · 1 citation

Bellman Policy Optimization

Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent, is introduced and it is proved that it has the same unique optimal solution as the original PMD objective.

Zhuo-Qing Song, Hao-Tian Xu, Xi-Kun Zhang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.