Skip to content

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

Sep 2026 · arXiv.org · Vol abs/2609.26749 · 0 citations · 21 references
Computer Science

TL;DR

It is argued that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and this is supported with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies.

Abstract

Large language models (LLMs) are increasingly applied to the automated repair of C/C++ security vulnerabilities, and compile rate is a commonly reported proxy for progress: whether the generated patch compiles. We argue that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and we support this with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies. Compile rate (i) barely responds to an intervention that substantially improves the generated code; (ii) is dominated by evaluation-harness and dataset artifacts rather than model quality, with about 64% of compile failures not attributable to the model, a share that is nearly invariant across models; (iii) shifts by 1.8 to 2.7 times on identical patches under a single compiler-standard flag, with zero regressions; (iv) ranks the three models in the opposite order to reference-similarity metrics; and (v) rewards non-repairs when used as an optimization target, since a compiler-feedback loop raises compile rate while similarity to the human fix falls, with manual inspection finding deletion- and placeholder-style non-repairs among the newly compiling outputs. The natural fallback, whole-function CodeBLEU, also fails: an unchanged copy of the vulnerable input outscores every model. We also examine diff_F1, a change-aware screen that scores only the edited region. It gives exactly zero credit to a no-op and near-zero credit to some, though not all, of the deletion-based gaming patches we observed, while still crediting genuine partial edits, so it may serve as a cheap screen before deeper, execution-based analysis. It is not a repair-quality metric, and we report where it falls short. Our findings argue for change-aware, execution-grounded evaluation of LLM-based vulnerability repair.

View source

Similar papers

Preprint Oct 2026

Beyond Functional Correctness: Static Code Quality Profiling of LLM-Generated Code

Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that...

Jue-Zhe Huang, Tarek Mahmud, Mei-Ru Che et al. · 0 citations
Preprint Oct 2026

On the Reliability of LLM-Based Vulnerability Patching Benchmarks

These results show that benchmark scores are highly sensitive to evaluation design, and practical guidelines for more rigorous, reliable, and reproducible evaluation are provided.

Dang K. Le, Wen-Xuan Shi, Xin-Yu Xing · 0 citations
#artificial intelligence Preprint Oct 2026

Characterizing Overconfident Failure in LLM-Based Code Generation

Large language models (LLMs) are increasingly used for automated code generation, but generated programs can appear syntactically plausible while still failing execution-based correctness checks. Existing validation methods, such as testing and program analysis, remain essential but are often incomplete, costly, or app...

Ravishka Rathnasuriya, Wei Yang · 3 citations
#large language models Open access Sep 2026

Fixing, Breaking, or Faking It? An Execution-Calibrated Evaluation of LLM Vulnerability Patching in JavaScript, Python, Go, and Java, and the Limits of LLM-as-Judge

Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness, leaving no single judge trustworthy.

Patrick Deininger, Wolfgang Slany · 1 citation
Open access Oct 2026

Inducing Code Change in LLM Self-Debugging: Effects on Program Repair

Execution-guided self-debugging can return unchanged code when revision is requested, and even a changed program may still fail to repair the observed failure. A final pass/fail result alone does not distinguish these different outcomes. We study them using 9790 Python programs generated by six open-weight models on th...

Jindae Kim · 0 citations
Open access Oct 2026

RICE: Harnessing LLMs and Historical Issues to Discover Internal Rust Compiler Errors

Rust is a modern system-level programming language that emphasizes safety, concurrency, and performance. Ensuring the reliability of the Rust compiler is critical, as undetected compiler defects, particularly internal compiler errors (ICEs), can lead to runtime crashes or undefined behavior in system software. Existing...

Lang-Yi Lu, Wei You, Bin Liang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.