Skip to content

The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration

Sep 2026 · arXiv.org · Vol abs/2609.33085 · 1 citation · 24 references
Computer Science

TL;DR

This work introduces Problem--Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is often limited by insufficient exploration: difficult problems can yield uniformly incorrect rollout groups and therefore little learning signal. We show that such failures need not reflect missing capability. Instead, finite sampling often concentrates on a problem-specific dominant reasoning strategy while leaving alternative strategies already supported by the model unexplored. Moreover, the accessibility of these strategies evolves during RL: some are internalized into autonomous behavior, while others become difficult to elicit before being absorbed. Motivated by these observations, we introduce Problem--Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups. A preservation objective keeps useful strategy-conditioned routes accessible while successful guided behaviors are transferred to the unguided policy. Across Qwen2.5 models from 1.5B to 7B and two RL training corpora, PSRA consistently improves reasoning performance, reduces dead saturation, strengthens out-of-distribution transfer, and maintains larger gains under increased inference budgets.

View source

Similar papers

Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR

GRAFT (Gated Replacement of Answer-Failed groups with peer Trajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups and transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level co...

Doohyuk Jang, Y. Park, Gyouk Chu et al. · 0 citations

Where the Model Changes Its Mind: Hindsight-Divergence Localization for Efficient Reinforcement Learning with Verifiable Rewards

Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choice...

Fan-Chao Chen, Heng-Yu Fu, Shivaram Venkataraman et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents

CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both Signal starvation and policy drift, internalizes long-horizon capability directly into a small open model; the complete training stack is planned to be released at https://github.com/AlibabaResearch/SignalCoverageRL.

Li-Ming Pu, Xiao-Xiao Li, Yi-Fu Liu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SUN: Reaching for Novelty in Reinforcement Learning

This paper proposes SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable and presents an adaptive goal-selection strategy that leverages these properties, and an accurate yet lightweight pseudocount to avoid the overhead of classic methods.

Wen-Yan Yang, A. Mustafin, Dominik Baumann et al. · 0 citations
Preprint Aug 2026

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

This work proposes two complementary strategies to improve the performance of value function RL: Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; and TETHER, a baseline that adaptively interpolates between gro...

S. Venkatraman, Matthieu Dinot, Laurence Aitchison · 1 citation · ⚡1

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.