Skip to content

Shared Experience, Separate Learning: Companion Confidence Calibration for LLMs

Sep 2026 · 0 citations · 51 references
Computer Science

TL;DR

CoCal is introduced, which trains a lightweight companion from rollout hidden states and verifier-derived correctness supervision while leaving task optimization unchanged, and shows that CoCal improves confidence estimation without sacrificing task performance, outperforming both RL-based concurrent methods and matched post-hoc calibration.

Abstract

Reliable self-assessment is essential for large language models (LLMs), yet they often remain highly confident when their answers are wrong. We study \emph{concurrent confidence calibration}, where confidence is learned alongside capability improvement rather than calibrated only after training. Reinforcement learning from verifiable rewards (RLVR) provides a natural setting for this paradigm, as it continuously produces responses paired with verifiable correctness feedback. Existing concurrent methods, however, learn both capability and confidence through reinforcement learning within shared policy parameters, potentially coupling two fundamentally different learning problems. We instead propose \emph{shared experience but separate learning}: capability and confidence learn from the same trajectories, but through separate optimization mechanisms and parameters. Based on this principle, we introduce \textbf{CoCal (Companion Confidence Calibration)}, which trains a lightweight companion from rollout hidden states and verifier-derived correctness supervision while leaving task optimization unchanged. Experiments on Qwen3-8B and Qwen3-14B show that CoCal improves confidence estimation without sacrificing task performance, outperforming both RL-based concurrent methods and matched post-hoc calibration. The learned companion further generalizes across domains and policy shifts, while the benefits of CoCal persist at both scales.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Test-time Calibration Learning for Large Language Model Reasoning

Reliable large language models (LLMs) must not only produce accurate answers but also express confidence that faithfully reflects their probability of being correct. Such calibration is essential for identifying uncertain predictions and supporting reliable decision-making in real-world deployment. Recent studies incor...

Zi-Zhuo Zhang, Xiong Peng, Jing-Wei Sun et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Learning to Reason with Selective Internal Feedback

Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF...

Xu-Zheng-Long-Wang Lu, Shu-Meng Yang, Xin-Rui Wang et al. · 0 citations
Preprint Sep 2026

Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

Graphical User Interface (GUI) grounding is essential for autonomous agents to map natural language instructions to precise screen coordinates. However, existing supervised fine-tuning and reinforcement learning methods are constrained by the high cost of annotation, creating a scalability bottleneck. In this paper, we...

Yi-Zhou Liu, Fei Tang, Yuchen Yan et al. · 0 citations
Conference Aug 2026

Post-Training for Reasoning LLMs with Reinforcement Learning: A Stability–Efficiency Perspective

Reinforcement Learning (RL)-based post-training has become a key mechanism for improving reasoning Large Language Models (LLMs), especially when automated verifiers enable Reinforcement Learning with Verifiable Reward (RLVR). Recent methods, however, are increasingly modular: many apparent algorithmic advances recombin...

Liu Yang, Han Zhu, Zheng-Yang Zhong et al. · 2 citations

OpenJev-RLCD: A Working RLCD Implementation

Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working impleme...

Zhi-Min Gao, Pi-Chao Wang · 0 citations
#artificial intelligence Preprint Sep 2026

Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts

CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression, turning the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together.

Chen-Xiao Fan, Chong-Ming Gao, Gang-Yi Zhang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.