Nov 2026· Proceedings of the 35th ACM International Conference on Information and Knowledge Management· 0 citations· 50 references
TL;DR
POLAR is presented, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner that can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle.
Abstract
Reinforcement learning with verifiable rewards (RLVR) is a practical way to improve the reasoning and decision-making abilities of large language models, but its training signal is often sparse. When a policy faces prompts beyond its current ability, every sampled trajectory in a group may fail the verifier, resulting in an all-zero group. In this case, group-relative policy optimization gives little useful gradient even when the prompt contains valuable learning content. This capacity-difficulty mismatch is especially costly for smaller models used in knowledge-intensive and web-interaction tasks. We present POLAR, a difficulty-aware training framework that augments on-policy RL with conditional guidance from a larger reasoner. POLAR uses the standard GRPO branch when at least one sampled response succeeds, and switches to a supervised objective over verified teacher trajectories only when the whole student group receives zero reward. The method can turn otherwise wasted all-zero groups into targeted learning opportunities while keeping policy exploration on prompts the student can handle. We formalize the switching criterion, teacher selection objective, mixed RL/supervised loss, and compute overhead. We evaluate POLAR on WebShop, GAIA, and WebWalkerQA using Qwen2.5-7B-Instruct, Qwen3-4B, and Qwen3-8B policies. The results show consistent gains over matched GRPO baselines, with large improvements on hard web traversal questions for the two smaller policies. Ablations and diagnostics further show that the gains come from the proposed all-zero recovery mechanism.
Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding...
Ze-Hao Wang, Han-Zhu Chen, Run-Quan Gui et al.· 1 citation
The findings support on-policy distillation as an effective warmup for agentic RLVR and identify initial reward discovery as a mechanism that can contribute to the observed acceleration.
Yi-Tong Qiao, Tian-Tian He, Lei Liu et al.· arXiv.org· 0 citations
Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize s...
Kevin Frans, Alexander M. Rush, Sergey Levine et al.· 0 citations
This work introduces Problem--Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups.
Jin Cui, Xin-Yue Long, Bo-Ran Zhao et al.· arXiv.org· 1 citation
This work theoretically links flat reward RLVR to reduced distribution sharpening, and empirically shows improved sample diversity while maintaining single-attempt accuracy, in a method called \texttt{Flat}-RLVR that searches for flat reward regions via sharpness-aware minimization during LLM post-training, combining t...
Yeonbum Kim, Hyun Lee, Sung Whan Yoon· 0 citations
Modern language models undergo reinforcement learning with verifiable rewards (RLVR) on top of already-trained checkpoints. A key promise of RLVR is the discovery of new reasoning strategies. In principle, a model can sample novel ideas absent from its prior training data. In practice, however, augmenting RLVR with str...
Saif Punjwani, Micah Goldblum· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT scientists developed a training method to generate models that simulate the effects of human activities on climate much more quickly than other approaches.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.