This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, dense feedback, off-policy reuse, and rollout–update dataflow.
Abstract
Reinforcement learning (RL) has become a central post-training approach for reasoning and agentic large language models (LLMs), particularly when task outcomes can be verified automatically. Comparisons across this literature remain difficult because a reported gain may combine changes to the learning signal, policy constraint, response granularity, data reuse, rollout system, and inference budget. This survey develops a mechanism-driven framework for separating these effects. It organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, dense feedback, off-policy reuse, and rollout–update dataflow. We use this framework to synthesize representative method families, analyze cross-motif interactions, and conduct source-bounded case studies of how multicomponent recipes and asynchronous pipelines should be interpreted. We further distinguish literature-established diagnostics from survey-defined reporting primitives and provide a matched-budget reporting checklist. Finally, we discuss how these interfaces may transfer to vision-language-action (VLA) models, World Action Models (WAMs), and embodied-agent post-training for unmanned systems, where reports should describe action representation, world-model error, interaction cost, and safety constraints. The corpus emphasizes mechanism clarity and records the maturity of evidence from recent preprints and system reports.
Large language models (LLMs) are increasingly described as acquiring"reasoning"skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analy...
Reinforcement learning with verifiable rewards (RLVR) is a practical way to improve the reasoning and decision-making abilities of large language models, but its training signal is often sparse. When a policy faces prompts beyond its current ability, every sampled trajectory in a group may fail the verifier, resulting...
Ye Yuan, Zhi-Ping Xiao, Ming Zhang· Proceedings of the 35th ACM...· 0 citations
This work proposes RLiD (Reinforcement Learning in the Decision plane), a framework that directly optimizes the probabilities of executable action options and avoids invalid action outputs and improves RL rollout efficiency by eliminating autoregressive token generation and bypassing action parsing.
Zi-Yi Chen, Yan Zhang, Jian-Hui Wei et al.· 0 citations
This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives.
N. Gholipour, M. Assunção, Gursimran Singh et al.· arXiv.org· 0 citations
Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding...
Ze-Hao Wang, Han-Zhu Chen, Run-Quan Gui et al.· 1 citation
Step-Advantage Policy Distillation (SAPD) is proposed, a method for learning from offline trajectories using relative step advantages, allowing the student to generate responses conditioned on recorded histories, and introduces relative step advantage signals to stabilize training.
Jun-Cheng Diao, Hai-Bin Wang, Wei Shi et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.