This paper proposes CARE, a method which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost.
Abstract
Reinforcement learning (RL) has proven effective in enhancing the reasoning performance of large language models (LLMs), particularly in complex mathematical and programming tasks. However, this capability comes with systematic \textit{length misallocation}, in which models devote excessive reasoning to simple questions while terminating prematurely on harder ones, degrading inference efficiency with negligible accuracy improvement. Many length-adaptive methods mitigate this issue by allocating token budgets according to question difficulty, under the implicit assumption that harder questions benefit monotonically from extended reasoning. In contrast, we find that the effect of reasoning length on accuracy is concentrated on \textit{partially solvable} questions. Our further analysis reveals that explicit length rewards can produce unintended training dynamics. Motivated by these findings, we propose \textbf{CARE}---\textbf{C}ontrastive \textbf{A}ccuracy \textbf{R}eward \textbf{E}stimation---which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost. Experiments across multiple reasoning benchmarks demonstrate that our method improves Pass@1 by up to \(4\%\) while simultaneously reducing reasoning length by \(37\%\), achieving higher token efficiency. Code will be available upon the acceptance of this paper.
A reasoning model is built that adaptively chooses how much to reason for each problem, and the brief modes end up more accurate than \textsc{Long}, which shows that the router sorts problems by difficulty rather than at random.
Gijs Kassenaar, Zhao Yang, Vincent François-Lavet· 2 citations
Rita is proposed as a novel RL paradigm that performs consistently superior to the supervised finetuning approaches and vanilla RL-finetuned frameworks and adopts a difficulty-aware data filtering strategy that selects informative easy-to-medium samples for RL using rollout error rate and reward variance.
Peng-Zhan Sun, Shiu-hong Kao, Shi-Jie Li et al.· 1 citation
This work takes inspiration from a resource-rational perspective on human cognition and introduces a novel GRPO reward that encourages efficient reasoning about whether the task contains all the information needed to solve it, leading to human-like abstention performance gains.
Polina Tsvilodub, M. Höth, Michael Franke et al.· 0 citations
Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad \textit{memorization bias}, where familiar content improves re...
Zineddine Tighidet, A. Mogini, Jia-Li Mei et al.· 0 citations
This work establishes a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution, and reveals that FEEs consistently yield performance improvements over standard settings.
This work proposes When2Think, an RLVR-based post-training framework for instance-adaptive computation allocation that requires neither a learned reward model nor a learned critic, and offline reference caching avoids online reference-model queries during policy updates.
Jaejun Shim, Hyunjin Kim, Young Jin Kim et al.· arXiv.org· 1 citation