Evaluating the Effectiveness of Repeated Self- Consistency Passes by Small Language Models for Binary Vulnerability Detection in Source Code

Abstract

This paper addresses stochastic variance in the decisions of large language models (LLMs) during binary classification of software vulnerabilities, using the open-source model Qwen3-8B. We evaluate the self-consistency method on a balanced Python dataset built for this purpose. The method's effectiveness depends strongly on the aggregation strategy, the number of passes K, and the model's error pattern. Under a selection rule that favors overall accuracy, Majority voting with 11 passes increased accuracy from 0.5422 to 0.5904 and F1 from 0.5615 to 0.6016. The improvement in paired correctness was statistically significant by McNemar's test, though the reduction in missed vulnerabilities (false negative rate, FNR) was modest under this setting. In contrast, two alternative strategies - Threshold voting and OR-voting - produced a larger FNR reduction at the cost of a predictable drop in accuracy. These results support flexible control over the precision-recall trade-off for security defect detection in DevSecOps workflows.