DTC is introduced, a framework that estimates confidence by counting tokens at which two models strongly disagree during decoding, and finds that their count is almost negatively associated with answer accuracy, thereby serving as a simple yet effective signal for uncertainty quantification.
Abstract
As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming increasingly important for quantifying the uncertainty of their answers. Current methods for estimating the confidence of large language models are generally based on probabilities of selected key tokens, but the underlying mechanism remains unclear. Our pilot study finds that replacing selected token probabilities with coarse substitutes can also improve calibration, motivating us to further explore effective signals of model confidence. We introduce Divergent Token Confidence (DTC), a framework that estimates confidence by counting tokens at which two models strongly disagree during decoding. DTC identifies these divergent tokens using the Jensen-Shannon divergence between next-token distributions evaluated along the same reasoning trajectory. We find that their count is almost negatively associated with answer accuracy, thereby serving as a simple yet effective signal for uncertainty quantification. DTC supports both white-box and black-box evaluation using auxiliary models, without explicit training and affecting the generation process. Experiments across multiple model families and six mathematical benchmarks demonstrate improved calibration over probability-based and verbalized baselines. Under white-box evaluation, the count-only estimator achieves an average expected calibration error of 13.0%, compared with 32.7%-42.4% for standard full-sequence confidence methods. In black-box settings, it also improves calibration over the original verbalized scores. For example, mean expected calibration error falls from 32.1%-40.2% to 13.7%-16.3% on DeepSeek-V3.2. These findings provide new insights for improving reasoning uncertainty quantification in large language models. The code is released at https://github.com/szu-tera/DTC.git.
A declarative probabilistic framework for pre-training evaluation that makes the semantic structure of model behavior explicit and provides new formal tools for relating evaluation to learning, and links evaluation and learning through a shared semantics.
Kyle Richardson, C. Anderson, Pranav Balakrishnan et al.· Proceedings of the 2026 Conf...· 1 citation
DirEAG is proposed, a Dirichlet Evidence Aggregation method that converts each elicited answer-confidence observation into calibrated soft evidence over generated candidate answers and an additional null state, allowing the model to represent cases where none of the candidates is correct.
The U-Space is introduced, a low-dimensional subspace that makes a model's evolving uncertainty measurable and interpretable and outperforms established baselines under both standard and length-controlled evaluation and transfers more reliably than supervised estimators.
Tobias Braun, Nils Loose, Alexander Herzog et al.· 0 citations
The Distributional Answer Commitment Score (DACS) is proposed, a distributional confidence estimator that measures the entropy of the model's probability distribution over answer commitment at each reasoning step to capture how concentrated the model's answer belief is without requiring ground-truth answers or task-spe...
Zhao-Han Zhang, Jun-Jie Liu, Chengzhengxu Li et al.· arXiv.org· 0 citations
This work presents a diagnostic and optimization framework grounded in a key empirical finding: the value of tokens within a CoT reasoning sequence is highly non-uniform, and this non-uniformity can be effectively characterized by token-level log probability signals.
Run-Jia Zeng, Hang Hua, Yiyang Liu et al.· arXiv.org· 1 citation
This work instantiate this framework with p, a hybrid estimator that combines verbal confidence with spectral answer diversity computed from the negative von Neumann entropy of sampled answer embeddings, and achieves higher average AUROC than verbal confidence and sampling-based uncertainty while using half as many sam...
Rounak Sharma, Ananya B. Sai, Soumyabrata Pal· 0 citations