CoCal is introduced, which trains a lightweight companion from rollout hidden states and verifier-derived correctness supervision while leaving task optimization unchanged, and shows that CoCal improves confidence estimation without sacrificing task performance, outperforming both RL-based concurrent methods and matched post-hoc calibration.
Abstract
Reliable self-assessment is essential for large language models (LLMs), yet they often remain highly confident when their answers are wrong. We study \emph{concurrent confidence calibration}, where confidence is learned alongside capability improvement rather than calibrated only after training. Reinforcement learning from verifiable rewards (RLVR) provides a natural setting for this paradigm, as it continuously produces responses paired with verifiable correctness feedback. Existing concurrent methods, however, learn both capability and confidence through reinforcement learning within shared policy parameters, potentially coupling two fundamentally different learning problems. We instead propose \emph{shared experience but separate learning}: capability and confidence learn from the same trajectories, but through separate optimization mechanisms and parameters. Based on this principle, we introduce \textbf{CoCal (Companion Confidence Calibration)}, which trains a lightweight companion from rollout hidden states and verifier-derived correctness supervision while leaving task optimization unchanged. Experiments on Qwen3-8B and Qwen3-14B show that CoCal improves confidence estimation without sacrificing task performance, outperforming both RL-based concurrent methods and matched post-hoc calibration. The learned companion further generalizes across domains and policy shifts, while the benefits of CoCal persist at both scales.
Reliable large language models (LLMs) must not only produce accurate answers but also express confidence that faithfully reflects their probability of being correct. Such calibration is essential for identifying uncertain predictions and supporting reliable decision-making in real-world deployment. Recent studies incor...
Zi-Zhuo Zhang, Xiong Peng, Jing-Wei Sun et al.· 0 citations
Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF...
Xu-Zheng-Long-Wang Lu, Shu-Meng Yang, Xin-Rui Wang et al.· 0 citations
Graphical User Interface (GUI) grounding is essential for autonomous agents to map natural language instructions to precise screen coordinates. However, existing supervised fine-tuning and reinforcement learning methods are constrained by the high cost of annotation, creating a scalability bottleneck. In this paper, we...
Yi-Zhou Liu, Fei Tang, Yuchen Yan et al.· 0 citations
Reinforcement Learning (RL)-based post-training has become a key mechanism for improving reasoning Large Language Models (LLMs), especially when automated verifiers enable Reinforcement Learning with Verifiable Reward (RLVR). Recent methods, however, are increasingly modular: many apparent algorithmic advances recombin...
Liu Yang, Han Zhu, Zheng-Yang Zhong et al.· International Conference on...· 2 citations
Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working impleme...
CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression, turning the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together.
Chen-Xiao Fan, Chong-Ming Gao, Gang-Yi Zhang et al.· 0 citations
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.