Beyond Bypass: Measuring Vulnerability Regression in LLM-Generated Security Patches

Abstract

Automated vulnerability repair with large language models is increasingly woven into security workflows, yet evaluation practice has focused almost entirely on a single question: does the patch eliminate the targeted weakness? However, an equally important question has received little systematic attention: has the patching act itself introduced new weaknesses into the function being repaired? Bypass testing and regression measurement are orthogonal properties, and neglecting the latter leaves a material blind spot in any assessment of patch quality.This paper presents an empirical study of vulnerability regression in LLM-generated security patches, applying differential static analysis to 479 complete C/C++ functions spanning eight CWE types. Patches are generated from three frontier models alongside developer-authored ground-truth fixes, for a total of 1,916 analyzed patch–function pairs. AI models fix targeted vulnerabilities at rates approaching human developers, yet they leave a meaningfully larger share of patches in a structurally neutral state, modifying code without engaging its underlying weakness. Across all three AI models, regression rule sets are more similar to each other than to human regression patterns, a consistency specific enough to target with shared automated validators. Finally, for human developers, successfully fixing a vulnerability is strongly associated with lower regression risk. However, this association is absent and not statistically detectable for AI patchers, a distinction with direct implications for how AI patch quality should be verified.