Skip to content

Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation

Nov 2025 · arXiv.org · Vol abs/2511.11500 · 7 citations · 94 references
Computer Science

TL;DR

These results establish abstention as a first-class training objective that transforms ``I don't know''from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.

Abstract

Modern language models fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, -$\lambda$ error) instead of binary. Controlled experiments on logic puzzles reveal that varying $\lambda$ produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. The same frontier holds on MATH Levels 4--5 and on medical QA, where it transfers to an unseen dataset. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms ``I don't know''from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.

View source

Similar papers

#natural language process... Preprint Oct 2026

CalibraRL: Confidence-Calibrated Reinforcement Learning for Mitigating Hallucinations in RAG

Large language models (LLMs) often produce plausible but incorrect answers, motivating reinforcement learning methods that reward correct answers, penalize incorrect answers, and allow abstention. However, fixed outcome-based rewards do not distinguish incorrect responses with different confidence levels or account for...

Qianchi Zhang, Hainan Zhang, Liang Pang et al. · 0 citations
#machine learning Preprint Oct 2026

Reinforcement Learning Done Right for Explicit Confidence Targets

Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their be...

Ze-Feng He, Yi-Cheng He, Yuan Lin et al. · 1 citation

The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models

Bounded margins mitigate confident hallucinations during post-training, implemented through an entropy-dependent margin bound in direct preference optimization (DPO) and shown to mitigate confident hallucinations during post-training.

Qing-Jia Huang, Ya-Kai Li, Jian-Guo Wu et al. · 0 citations
#machine learning Preprint Sep 2026

ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

ImpossibleRubrics is introduced, a benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate specifying what an honest answer may and may not claim, together with 48 answerable controls.

Bo-Wen Qin, Yi Xie, Ye-Sheng Liu et al. · 2 citations

Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection

A useful language model should know when to answer and when to say"I don't know."Can we improve that decision by training the model itself or by training a separate component to check its answers? We study both approaches on Qwen3-14B for factual questions, using a score that rewards correct answers, penalizes mistakes...

Juan R. Terven, Diana-Margarita Córdova-Esparza, R. Gómez et al. · 0 citations
#machine learning Preprint Oct 2026

CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty

Large language models (LLMs) are becoming prevalent in scenarios where honesty is critical. However, despite alignment training, these models still exhibit dishonest behaviour, such as outputting claims that contradict some knowledge or are not supported by what is accessible in context. We introduce Continuous Adversa...

Yuan-Chen Chang, Niklas Herbster, G. Gidel et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.