Skip to content

Learning Process Rewards via Reasoning State Propagation

Sep 2026 · arXiv.org · Vol abs/2609.39220 · 0 citations · 34 references
Computer Science

TL;DR

Reasoning State Propagation (RSP) is introduced, which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory, allowing process annotations to supervise intermediate states while outcome labels supervise the final state.

Abstract

Process reward models (PRMs) have demonstrated notable effectiveness in test-time scaling and reinforcement learning by providing fine-grained signals for evaluating intermediate reasoning states, but their training relies heavily on costly process annotations. A natural way to alleviate this dependence is to complement limited process supervision with scalable outcome supervision. However, existing PRMs often model reasoning prefixes independently, providing no explicit mechanism for effectively using final outcome to guide the learning of intermediate reasoning states. We introduce Reasoning State Propagation (RSP), which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory. Specifically, RSP predicts a break probability that a valid state becomes invalid and a repair probability that an invalid state returns to valid. By propagating these transitions, RSP connects intermediate states to the final state, allowing process annotations to supervise intermediate states while outcome labels supervise the final state and can provide learning signals to preceding steps. Across reasoning search, response selection, and reinforcement learning, RSP consistently outperforms representative PRM baselines, with average improvements over Qwen2.5-Math-PRM of 5.6% in beam search and 2.1% in reinforcement learning.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Dependency-Aware Reward Shaping for Agentic Reinforcement Learning

When training large language models with reinforcement learning, terminal rewards provide little guidance about which steps matter. Common methods for assigning step credit overlook that work built on uncorrected mistakes is wasted while independent work remains valid. With only a final success/failure reward, every st...

Zi-Yi Chen, Yan Zhang, Jian-Hui Wei et al. · 1 citation · ⚡1

Inducing Process Supervision from Outcome-Only Reinforcement Learning

TIPS (Thinking-Induced Process Supervision), an outcome-only reinforcement learning (RL) framework for training generative PRMs, is introduced and validated, demonstrating the effectiveness of TIPS across math and agent benchmarks and four backbone families.

Sheng-Da Fan, Xin Cong, Zhong Zhang et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation

Reinforcement learning with verifiable rewards (RLVR) implemented with Group Relative Policy Optimization (GRPO) has driven substantial progress in training reasoning language models. However, verifiable outcome rewards provide an imperfect measure of reasoning quality: a correct final answer may not be supported by th...

Hai-Feng Qian, David Mosallanezhad, Christian Munley et al. · 0 citations
#machine learning Preprint Nov 2026

Plug-and-Play Verifier for Decoupling Process Reward in Stabilizing Long-Horizon Agentic Reasoning

Stabilizing long-horizon reasoning requires intelligent agents to sustain useful progress across successive reasoning steps and tool interactions. Such agents are typically trained with reinforcement learning (RL), where actor-critic RL uses a learnable critic network to estimate step advantages, while critic-free meth...

Ming-Wei Xu, Jie-Yuan Pei, Bing-Hao Tu et al. · 0 citations
Preprint Oct 2026

Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay

Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning capabilities of large language models, but often reduces the diversity of their reasoning behaviors. We study this phenomenon from the perspective of finite-sample learning dynamics and show that limited on-policy rollouts can progressively su...

Wen-Zhi Li, Sainyam Galhotra, C. Cardie · 0 citations
Preprint Oct 2026

ClueFlow: Learning Long-Horizon Search through Attributed Research-State Transitions

Long-horizon search requires agents to acquire relevant evidence and integrate it into an evolving research state. Existing methods maintain research states to guide exploration or assign turn-level progress signals, yet grounding these signals in explicit, attributable state changes remains underexplored. We introduce...

Yi-Cheng Wang, Wen-Tao Tan, Yi-Dan Huang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.