Skip to content

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

Sep 2026 · arXiv.org · Vol abs/2609.36932 · 0 citations · 30 references
Computer Science

TL;DR

This work proposes FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning, and introduces an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity.

Abstract

Recently, Group Relative Policy Optimization (GRPO) and its variants have been developed for policy optimization and demonstrated notable performance gains. However, these methods usually incur substantial computational overhead due to per-question multi-rollout sampling and repeated per-token probability evaluation across rollouts. Furthermore, low-information or highly homogeneous trajectories can degrade downstream learning signal efficiency, hindering model optimization and limiting final performance. To address these issues, we propose FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning. Specifically, 1) We introduce an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity. 2) Then, we design an adaptive rollout sampling mechanism to dynamically adjust the sampling scale across different training stages based on historical pruning distributions, balancing exploration adequacy and computational efficiency. Experiments demonstrate that FastRL can be seamlessly integrated into GRPO, DAPO, and GSPO variants, achieving an average 2.07$\times$ training speedup on Geometry3K and GeoQA8K-R1V, along with an approximately 1.64\% improvement in average accuracy on visual reasoning benchmarks. Source codes will be available at https://github.com/Nicozwy/FastRL.

View source

Similar papers

#machine learning Preprint Aug 2026

Group Adaptive Clipping Policy Optimization

Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage, is proposed, motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update he...

Sheng Jia, Xiao Wang, S. Kasiviswanathan et al. · 0 citations
#artificial intelligence Preprint Oct 2026

CERO: Where and When to Allocate Rollouts for RL Post-Training

Adaptive rollout methods for group-relative reinforcement learning typically allocate a fixed per-update budget across prompts. We instead study how to coordinate a finite rollout budget over the entire training horizon. We formulate this problem using a concave surrogate utility of cumulative prompt exposure and intro...

Yi-Ming Zong, Yi-Ge Wang, Xin-Ting Hu et al. · 0 citations
#machine learning Preprint Sep 2026

Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning

Dynamic Trajectory Valuation is proposed, a simple and efficient framework that estimates trajectory utility at the mini-batch level and filters detrimental trajectories based solely on gradient information by operating at the optimization level.

Xue-Song Jia, Ziao Yang, Zhan-He Huang et al. · 0 citations
#artificial intelligence Preprint Aug 2026

Learning from Hard Prompts: Difficulty-aware Advantage Amplification in Dynamic Sampling

Direct Advantage Amplification (DAA), which amplifies the advantages of hard-to-sample correct responses on hard prompts, as obtained by Dynamic Sampling, is proposed, which ensures that, when Dynamic Sampling is used, these hard-to-sample responses can be effectively capitalized on, implying higher training efficiency...

Si-Yuan Gan, Yu-Hang Li, Xi-Ran Wang et al. · 0 citations

Towards Better Training Signal: Advantage Clipped Policy Optimization

ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs, and establishes a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process.

Rui-Chuan Huang, Jing-Han Liu, Cong-Liang Chen · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.