It is argued that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and this is supported with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies.
Abstract
Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.
Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that...
Jue-Zhe Huang, Tarek Mahmud, Mei-Ru Che et al.· 0 citations
These results show that benchmark scores are highly sensitive to evaluation design, and practical guidelines for more rigorous, reliable, and reproducible evaluation are provided.
Dang K. Le, Wen-Xuan Shi, Xin-Yu Xing· 0 citations
Large language models (LLMs) are increasingly used for automated code generation, but generated programs can appear syntactically plausible while still failing execution-based correctness checks. Existing validation methods, such as testing and program analysis, remain essential but are often incomplete, costly, or app...
Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness, leaving no single judge trustworthy.
Patrick Deininger, Wolfgang Slany· Journal of Cybersecurity and...· 1 citation
Execution-guided self-debugging can return unchanged code when revision is requested, and even a changed program may still fail to repair the observed failure. A final pass/fail result alone does not distinguish these different outcomes. We study them using 9790 Python programs generated by six open-weight models on th...
Rust is a modern system-level programming language that emphasizes safety, concurrency, and performance. Ensuring the reliability of the Rust compiler is critical, as undetected compiler defects, particularly internal compiler errors (ICEs), can lead to runtime crashes or undefined behavior in system software. Existing...
Lang-Yi Lu, Wei You, Bin Liang et al.· Proceedings of the ACM on So...· 0 citations