Skip to content
#small language model Book Open access

POLAR: Policy Optimization through Larger Reasoners

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 50 references

TL;DR

POLAR is presented, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner that can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle.

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a practical way to improve the reasoning and decision-making abilities of large language models, but its training signal is often sparse. When a policy faces prompts beyond its current ability, every sampled trajectory in a group may fail the verifier, resulting in an all-zero group. In this case, group-relative policy optimization gives little useful gradient even when the prompt contains valuable learning content. This capacity-difficulty mismatch is especially costly for smaller models used in knowledge-intensive and web-interaction tasks. We present POLAR, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner. POLAR uses the standard GRPO branch when at least one sampled response succeeds, and switches to a supervised objective over verified teacher trajectories only when the whole student group receives zero reward. The method can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle. We formalize the switching criterion, teacher selection objective, mixed RL/supervised loss, and compute overhead. We evaluate POLAR on WebShop, GAIA, and WebWalkerQA using Qwen2.5-7B-Instruct, Qwen3-4B, and Qwen3-8B policies. The results show consistent gains over matched GRPO baselines, with large improvements on hard web traversal questions for the two smaller policies. Ablations and diagnostics further show that the gains come from the proposed all-zero recovery mechanism.

Read PDF

Similar papers

#artificial intelligence Preprint Oct 2026

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding...

Ze-Hao Wang, Han-Zhu Chen, Run-Quan Gui et al. · 1 citation
#machine learning Preprint Nov 2026

Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning

Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize s...

Kevin Frans, Alexander M. Rush, Sergey Levine et al. · 0 citations

The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration

This work introduces Problem--Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups.

Jin Cui, Xin-Yue Long, Bo-Ran Zhao et al. · 1 citation
Preprint Oct 2026

Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs

This work theoretically links flat reward RLVR to reduced distribution sharpening, and empirically shows improved sample diversity while maintaining single-attempt accuracy, in a method called \texttt{Flat}-RLVR that searches for flat reward regions via sharpness-aware minimization during LLM post-training, combining t...

Yeonbum Kim, Hyun Lee, Sung Whan Yoon · 0 citations
#artificial intelligence Preprint Oct 2026

Decoupling Exploration from Optimization in RLVR

Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with str...

Saif Punjwani, Micah Goldblum · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.