Skip to content
Preprint

RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs

Nov 2026 · 0 citations · 28 references
Computer Science

TL;DR

This work proposes RLiD (Reinforcement Learning in the Decision plane), a framework that directly optimizes the probabilities of executable action options and avoids invalid action outputs and improves RL rollout efficiency by eliminating autoregressive token generation and bypassing action parsing.

Abstract

Reinforcement learning (RL) has improved the capabilities of large language models (LLMs) across a wide range of tasks. However, token-level RL assigns an action's advantage to the entire generated response, including tokens specifying the parsed action, reasoning chains, and malformed content. Although the environment executes only the parsed action, variation in how the action is expressed in natural language can introduce additional variance into policy-gradient estimates. We propose RLiD (Reinforcement Learning in the Decision plane), a framework that directly optimizes the probabilities of executable action options. RLiD compiles observations, an optional compact memory, and a labeled action menu into the model input, and uses a large decision model (LDM) to select an action in a single forward pass. By eliminating autoregressive token generation and bypassing action parsing, RLiD avoids invalid action outputs and improves RL rollout efficiency. Our analysis shows that, within a fixed text policy, marginalizing over alternative realizations of the same action removes a zero-mean gradient component without changing the expected update. We further distill trained LDM policies into generative LLMs through supervision on their action trajectories. Experiments on ARC, Countdown, and Sokoban demonstrate improved performance or lower computational cost across several settings. On ALFWorld, the complete training and distillation pipeline reaches 99.3% success in 12.7 H100 GPU-hours, compared with 86.6% in 17.3 GPU-hours for token-level RL of the same LLM. With compiled memory, our strongest Qwen2.5-1.5B LDM and distilled generative LLM achieve 100% success on all 134 unseen and 140 seen ALFWorld evaluation games.

View source

Similar papers

Review Open access Sep 2026

Reinforcement Learning Post-Training for Reasoning Large Language Models: Methods, Systems, and Evaluation

This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, den...

Liu Yang, Han Zhu, Zheng-Yang Zhong et al. · 1 citation

Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for embodied intelligence, but fine-tuning them with reinforcement learning (RL) remains constrained by the cost of real-world robot interaction. Model-based reinforcement learning (MBRL) reduces this cost by using a learned world model to generate...

Yi-Fei Sheng, Hao-Xiang Ren, Zhi-Long Zhang et al. · 0 citations
#small language model Book Open access Nov 2026

POLAR: Policy Optimization through Larger Reasoners

POLAR is presented, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner that can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle.

Ye Yuan, Zhi-Ping Xiao, Ming Zhang · 0 citations
#artificial intelligence Preprint Oct 2026

Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

Rollout-Time Intervention is introduced, a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary.

Xi Chen, Rui-Xue Ding, Zhigaoyuan Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

DeciCAD: Harness-Grounded Learning of Action Policies for CAD Agents

Large Language Model agents operate through closed-loop action policies whose decision space is defined by an agent harness. We investigate how this operating space shapes agent behavior and policy learning in computer-aided design (CAD), a professional domain characterized by persistent state, geometric constraints, a...

Yu-Feng Peng, Yi-Xin Zhang, Zi-Yi Liu et al. · 0 citations
#machine learning Preprint Oct 2026

Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers

Reinforcement learning (RL) has become a standard tool for post-training language models on reasoning tasks, where the policy is updated by reward feedback while exploring the space of responses. Despite its empirical success, theoretical understanding of RL post-training remains limited, in particular of why on-policy...

Naoki Nishikawa, Taiji Suzuki · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.