Harnessing Uncertainty in Code Language Models: Lessons from Vulnerability Detection

Abstract

Code Language Models (CLMs) are increasingly adopted for vulnerability detection, yet knowing when to trust their predictions remains an open challenge. Uncertainty estimation has improved reliability in traditional discriminative models, but whether these benefits transfer to the generative paradigm of CLMs is unclear. This experience paper presents a systematic empirical study of uncertainty estimation in CLM-based vulnerability detection, evaluating four calibration methods and two uncertainty-driven enhancement frameworks across eight models (2B--16B parameters) under both in-distribution and out-of-distribution settings on 1,050 real-world projects. Our study reveals that: (1) calibration effectiveness is model-scale dependent: temperature scaling and ensembles deliver consistent gains, while Monte Carlo Dropout and Bayesian Low-Rank Adaptation by Backpropagation only benefit medium-scale models (approximately 6B--7B parameters); (2) uncertainty-based correction requires label-conditional modeling to be effective (up to 12% Macro-F1 improvement on out-of-distribution data), as global strategies show negligible effects; (3) curriculum learning fails to improve upon standard fine-tuning, as uncertainty-based difficulty ordering does not align with the learned decision boundary, causing training instability or negligible gains. These findings challenge assumptions about the universal transferability of uncertainty methods and provide practical guidelines for deploying uncertainty-aware vulnerability detection.