Skip to content
Review Open access

Reinforcement Learning Post-Training for Reasoning Large Language Models: Methods, Systems, and Evaluation

Sep 2026 · Unmanned Systems · 1 citation

TL;DR

This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, dense feedback, off-policy reuse, and rollout–update dataflow.

Abstract

Reinforcement learning (RL) has become a central post-training approach for reasoning and agentic large language models (LLMs), particularly when task outcomes can be verified automatically. Comparisons across this literature remain difficult because a reported gain may combine changes to the learning signal, policy constraint, response granularity, data reuse, rollout system, and inference budget. This survey develops a mechanism-driven framework for separating these effects. It organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, dense feedback, off-policy reuse, and rollout–update dataflow. We use this framework to synthesize representative method families, analyze cross-motif interactions, and conduct source-bounded case studies of how multicomponent recipes and asynchronous pipelines should be interpreted. We further distinguish literature-established diagnostics from survey-defined reporting primitives and provide a matched-budget reporting checklist. Finally, we discuss how these interfaces may transfer to vision-language-action (VLA) models, World Action Models (WAMs), and embodied-agent post-training for unmanned systems, where reports should describe action representation, world-model error, interaction cost, and safety constraints. The corpus emphasizes mechanism clarity and records the maturity of evidence from recent preprints and system reports.

Read PDF

Similar papers

#natural language process... Preprint Oct 2026

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

Large language models (LLMs) are increasingly described as acquiring"reasoning"skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analy...

Aishik Nagar, Cheston Tan · 0 citations
#reinforcement learning Book Open access Nov 2026

POLAR: Policy Optimization through Larger Reasoners

Reinforcement learning with verifiable rewards (RLVR) is a practical way to improve the reasoning and decision-making abilities of large language models, but its training signal is often sparse. When a policy faces prompts beyond its current ability, every sampled trajectory in a group may fail the verifier, resulting...

Ye Yuan, Zhi-Ping Xiao, Ming Zhang · 0 citations
Preprint Nov 2026

RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs

This work proposes RLiD (Reinforcement Learning in the Decision plane), a framework that directly optimizes the probabilities of executable action options and avoids invalid action outputs and improves RL rollout efficiency by eliminating autoregressive token generation and bypassing action parsing.

Zi-Yi Chen, Yan Zhang, Jian-Hui Wei et al. · 0 citations
#artificial intelligence Review Sep 2026

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives.

N. Gholipour, M. Assunção, Gursimran Singh et al. · 0 citations
#artificial intelligence Preprint Oct 2026

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding...

Ze-Hao Wang, Han-Zhu Chen, Run-Quan Gui et al. · 1 citation
#artificial intelligence Preprint Oct 2026

SAPD: Learning from Agent Experience without Environment Interaction

Step-Advantage Policy Distillation (SAPD) is proposed, a method for learning from offline trajectories using relative step advantages, allowing the student to generate responses conditioned on recorded histories, and introduces relative step advantage signals to stabilize training.

Jun-Cheng Diao, Hai-Bin Wang, Wei Shi et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.