Skip to content
Preprint

Same Beliefs, Different Policies: Learning Reusable Representations for Early Stopping in LLM Reasoning

Oct 2026 · 0 citations
Computer Science

TL;DR

BELT (BELief-based Termination), which separates outcome prediction from stopping control from end-to-end controllers, and its accuracy-cost trade-offs are competitive with or better than those of end-to-end controllers.

Abstract

Reasoning LLMs often continue reasoning after reaching a correct answer, wasting computation and sometimes replacing it with an incorrect one. Deciding when to stop is a metareasoning problem: the model must weigh what further reasoning could achieve against its cost, given what it can already answer. Direct utility training learns a stopping representation at a chosen token price; supporting new prices can require additional controller training. We introduce BELT (BELief-based Termination), which separates outcome prediction from stopping control. It learns beliefs about current and future answers from a frozen LLM's hidden states, then fits a linear stopping policy on the frozen features. The prediction targets are independent of token price, so one probe supports different accuracy-cost trade-offs through CPU policy refitting or threshold adjustment. Averaged over five out-of-distribution (OOD) math benchmarks, BELT reduces generated tokens by 41% on Qwen3-1.7B and 51% on OLMo-3-7B-Think without loss of accuracy. The same math-trained BELT transfers to coding and agentic tasks without training on code or agentic data. In all three domains, its accuracy-cost trade-offs are competitive with or better than those of end-to-end controllers. On OLMo3, its probe has 21.2x fewer parameters than one checkpoint of the strongest baseline, OS-Pruner, and needs 55x fewer training GPU-hours than OS-Pruner's five price-specific checkpoints.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding...

Ze-Hao Wang, Han-Zhu Chen, Run-Quan Gui et al. · 1 citation
Preprint Nov 2026

Not All Reasoning Generalizes: A Case Study on Learning Generalizable Reasoning Strategies in RLVR

Reinforcement learning with verifiable rewards (RLVR) can enable generalizable improvement for large language model (LLM) reasoning. This paper studies how to incentivize such generalization via reasoning strategies. We study this question with a controllable equation-solving framework: finding multiple inputs x for eq...

Zi-Qi Pang, Yu-Xiong Wang · 0 citations
#artificial intelligence Preprint Oct 2026

Expanding LLM Reasoning

Extra inference compute is usually spent on sampling more reasoning chains. We study where inside an existing chain an additional continuation should begin. We define expansion utility, the change in correctness from restarting a chain at a stored step, and measure it at every eligible step for nine models on six bench...

Rian Atri, Evan Luo · 0 citations

Learned Reporting Preferences in RLVR Can Conflict with the Current Request

Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using automatically checked answers. Yet convention-matched evaluation cannot reveal whether reinforcing one reporting convention reduces adherence to a different request tha...

Yu-Peng Chang, Wen-Xuan Zhang, Yuan Wu · 1 citation
#artificial intelligence Preprint Oct 2026

PORA: Learning When More Reasoning Pays Off for LLM Agents

Paired Outcome-guided Reasoning Allocation (PORA), which learns this decision by comparing short and long reasoning from the same interaction state by comparing their final quality and total processing to estimate when longer reasoning is valuable.

Bo Yan, Qi Zhang, Zhen Tan et al. · 0 citations
#machine learning Preprint Oct 2026

Reinforcement Learning Done Right for Explicit Confidence Targets

Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their be...

Ze-Feng He, Yi-Cheng He, Yuan Lin et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.