This work proposes FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning, and introduces an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity.
Abstract
Recently, Group Relative Policy Optimization (GRPO) and its variants have been developed for policy optimization and demonstrated notable performance gains. However, these methods usually incur substantial computational overhead due to per-question multi-rollout sampling and repeated per-token probability evaluation across rollouts. Furthermore, low-information or highly homogeneous trajectories can degrade downstream learning signal efficiency, hindering model optimization and limiting final performance. To address these issues, we propose FastRL, a novel plug-and-play reinforcement learning framework that simultaneously improves training efficiency and the effectiveness of policy learning. Specifically, 1) We introduce an advantage-aware pruning strategy to selectively preserve high-advantage trajectories while maximizing inter-trajectory gradient diversity. 2) Then, we design an adaptive rollout sampling mechanism to dynamically adjust the sampling scale across different training stages based on historical pruning distributions, balancing exploration adequacy and computational efficiency. Experiments demonstrate that FastRL can be seamlessly integrated into GRPO, DAPO, and GSPO variants, achieving an average 2.07$\times$ training speedup on Geometry3K and GeoQA8K-R1V, along with an approximately 1.64\% improvement in average accuracy on visual reasoning benchmarks. Source codes will be available at https://github.com/Nicozwy/FastRL.
Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage, is proposed, motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update he...
Sheng Jia, Xiao Wang, S. Kasiviswanathan et al.· 0 citations
DIEM is proposed, a principled and fully automated framework that makes data utilization adaptive throughout RFT and consistently outperforms strong static and dynamic baselines.
Haoru Tan, Sitong Wu, Yan-Feng Chen et al.· 0 citations
Adaptive rollout methods for group-relative reinforcement learning typically allocate a fixed per-update budget across prompts. We instead study how to coordinate a finite rollout budget over the entire training horizon. We formulate this problem using a concave surrogate utility of cumulative prompt exposure and intro...
Yi-Ming Zong, Yi-Ge Wang, Xin-Ting Hu et al.· 0 citations
Dynamic Trajectory Valuation is proposed, a simple and efficient framework that estimates trajectory utility at the mini-batch level and filters detrimental trajectories based solely on gradient information by operating at the optimization level.
Xue-Song Jia, Ziao Yang, Zhan-He Huang et al.· 0 citations
Direct Advantage Amplification (DAA), which amplifies the advantages of hard-to-sample correct responses on hard prompts, as obtained by Dynamic Sampling, is proposed, which ensures that, when Dynamic Sampling is used, these hard-to-sample responses can be effectively capitalized on, implying higher training efficiency...
Si-Yuan Gan, Yu-Hang Li, Xi-Ran Wang et al.· 0 citations
ACPO is a practical and effective alternative to conventional IS-ratio clipping for RL post-training of LLMs, and establishes a connection between ACPO and gradient clipping in policy mirror descent (PMD), which is a standard technique to stabilize optimization process.