Skip to content

From Outcomes to Strategies: Learning Strategy Utility for Mathematical Reasoning

Sep 2026 · 0 citations
Computer Science

TL;DR

This paper introduces SURE, a framework for learning and leveraging relative strategy utility, which is defined as the likelihood that a strategy supports a correct downstream solution under a given executor.

Abstract

Reinforcement learning with verifiable rewards has substantially improved mathematical reasoning. However, terminal correctness alone provides limited insight into the quality of high-level strategies, such as theorem selection and subgoal decomposition, when considered separately from their subsequent execution. This paper studies strategy utility, which is defined as the likelihood that a strategy supports a correct downstream solution under a given executor. We introduce SURE, a framework for learning and leveraging relative strategy utility. In this framework, high-level strategies are separated from their detailed reasoning. Based on the pairwise preferences constructed from strategy-conditioned rollouts and teacher-generated contrasts, a Strategy Reward Model is learned to estimate relative strategy utility. During reinforcement learning, the frozen reward model reads only the extracted strategy, whose score is combined with the correctness and format rewards in a sequence-level GRPO objective. Compared with outcome-and-format GRPO baselines, experiments show that SURE improves average pass@1 by 1.87%, 2.64%, and 2.93% across three policy backbones. Our method also achieves competitive or better accuracy than stronger reward baselines while requiring substantially lower GRPO-stage compute.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Improving Math Reasoning through Value-guided Informative Search

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the mathematical reasoning capabilities of large language models. Recent work introduces search into RLVR rollouts to increase trajectory diversity, but diversity alone does not ensure that the search-induced rollout policy improves upon t...

Shao-Huai Liu, Yu-Ning Wu, Hao Liu et al. · 0 citations

The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration

This work introduces Problem--Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups.

Jin Cui, Xin-Yue Long, Bo-Ran Zhao et al. · 1 citation
#machine learning Preprint Oct 2026

Function-Structured Reinforcement Learning with Executable Verifiers for Mathematical Reasoning

Algorithmic mathematical reasoning requires reliable decomposition, computation, and aggregation. Final-answer rewards provide limited guidance on intermediate errors, while successful execution does not guarantee mathematical correctness. This work proposes Function-Structured Graph Reinforcement Learning (FSG-RL), co...

Zi-Han Liu, Xu-Rong Xie · 0 citations
#natural language process... Preprint Sep 2026

ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying

This work develops a new reinforcement learning signal named ConsensusPR, which directly reduces the reward sparsity of outcome reward across long reasoning trajectories, and introduces ConsensusBench, a novel dataset designed to provide rule-based process-level signals.

Shi-Qi Yan, Chao-Hong Tan, Qian Chen et al. · 0 citations
#machine learning Preprint Sep 2026

Cliff: Learning Process Rewards from the First Mistake

Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout, is proposed and established as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.

Pei-Xuan Han, Runnan Wang, Ketan Ramaneti et al. · 1 citation
#machine learning Preprint Sep 2026

CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards

CRS is introduced, a selection signal derived from 46 reasoning-sensitive attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts, which runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces bet...

Zhuo-Fan Chen, Zi-Qian Jiao, Yi-Kai Cui et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.