Skip to content

Towards Better Training Signal: Advantage Clipped Policy Optimization

Sep 2026 · arXiv.org · Vol abs/2609.36816 · 0 citations · 27 references
Computer Science

TL;DR

ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs, and establishes a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process.

Abstract

Reinforcement learning (RL) has become a cornerstone for improving the reasoning capabilities of large language models (LLMs), but the need for on-policy data substantially limits training efficiency. Reusing off-policy data through importance sampling (IS) can improve efficiency but introduce considerable instability. Hence, algorithms such as PPO and GRPO widely adopt IS-ratio clipping to stabilize training. However, training stability and gradient estimate are mainly determined by the product of IS ratio and advantage. To further stabilize training, we propose ACPO, which clips the product of the IS ratio and the advantage, leading to more stable gradient estimates. We also establish a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process, and prove the convergence of clipped-PMD under the standard RL setting. Experiments on widely used mathematical reasoning benchmarks show that ACPO consistently outperforms PPO and GRPO in both accuracy and training efficiency, delivering 4-6 percentage points gains on standard math benchmarks, with Qwen3-8B+PPO. Hence, ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs.

View source

Similar papers

Preprint Nov 2026

Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization...

Shui-Bai Zhang, Junhyuck Kim, Gyeongman Kim et al. · 0 citations

Score Centering Stabilizes Off-policy Reinforcement Learning

It is shown that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step, and an additive score centering term is derived that stabilizes RL under TIM by canceling drift.

Martina Marek, Max Ryabinin · 11 citations · ⚡3
#small language model Book Open access Nov 2026

POLAR: Policy Optimization through Larger Reasoners

POLAR is presented, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner that can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle.

Ye Yuan, Zhi-Ping Xiao, Ming Zhang · 0 citations
#machine learning Preprint Oct 2026

Reinforcement Learning via Advantage Guidance

Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems suffer from instability and hyperparameter sensitivity. In contrast, supervised learning provides a stable and scalable training objective, but is fundamentally limited by...

Zhi-Yuan Zhou, Sergey Levine · 0 citations
Preprint Oct 2026

Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs

This work theoretically links flat reward RLVR to reduced distribution sharpening, and empirically shows improved sample diversity while maintaining single-attempt accuracy, in a method called \texttt{Flat}-RLVR that searches for flat reward regions via sharpness-aware minimization during LLM post-training, combining t...

Yeonbum Kim, Hyun Lee, Sung Whan Yoon · 0 citations
#machine learning Preprint Oct 2026

Baselines and Score Centering in Policy Gradient Estimation

Reinforcement learning has become a widely used approach to post-training large language models. In a typical implementation, learning is off-policy: The policy generating the responses differs from the policy being trained because of training-inference mismatch or policy staleness in asynchronous training. This discre...

Shu-Yi Han, Zi-Qing Hu · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.