Benchmarking Automated Security Patch Backporting: How Far Are We?

Beyond Bypass: Measuring Vulnerability Regression in LLM-Generated Security Patches

Automated vulnerability repair with large language models is increasingly woven into security workflows, yet evaluation practice has focused almost entirely on a single question: does the patch eliminate the targeted weakness? However, an equally important question has received little systematic attention: has the patching act itself introduced new weaknesses into the function being repaired? Bypass testing and regression measurement are orthogonal properties, and neglecting the latter leaves a material blind spot in any assessment of patch quality.This paper presents an empirical study of vulnerability regression in LLM-generated security patches, applying differential static analysis to 479 complete C/C++ functions spanning eight CWE types. Patches are generated from three frontier models alongside developer-authored ground-truth fixes, for a total of 1,916 analyzed patch–function pairs. AI models fix targeted vulnerabilities at rates approaching human developers, yet they leave a meaningfully larger share of patches in a structurally neutral state, modifying code without engaging its underlying weakness. Across all three AI models, regression rule sets are more similar to each other than to human regression patterns, a consistency specific enough to target with shared automated validators. Finally, for human developers, successfully fixing a vulnerability is strongly associated with lower regression risk. However, this association is absent and not statistically detectable for AI patchers, a distinction with direct implications for how AI patch quality should be verified.

Benchmarking MCP Vulnerability Detection from a Paired Evaluation Perspective

Model Context Protocol (MCP) servers have become essential for connecting large language models (LLMs) with external tools, data sources, and execution environments. However, their privileged access and bridging role introduce significant security risks. Despite many LLM-based detection methods, it remains unclear whether they can semantically differentiate paired vulnerable and fixed MCP servers rather than just identifying risky API calls. In this work, we present a dedicated benchmark for pairing MCP server vulnerability detection, comprising 75 paired vulnerable and fixed MCP tools synthesized by Gemini-1.5-Pro and manually cross-validated. We adopt the Matched Pair Accuracy (MPA) metric to evaluate detection methods. Claude Haiku-4.5 achieves the best MPA of 0.853, while GPT-5-mini attains 0.842 F1 but only 0.653 MPA, showing that conventional metrics can overestimate paired detection performance. Our benchmark provides a new perspective for systematic, reproducible, and MCP-specific security evaluation.

SecVulEval: Context-Aware Benchmarking of LLMs for Vulnerability Detection

Large Language Models (LLMs) show promise for vulnerability detection, but their evaluation is limited by the lack of high-quality benchmarks. Most existing datasets rely on coarse function-level labels, overlook fine-grained vulnerability patterns, and lack critical program context such as data/control dependencies. They also suffer from data quality issues, including mislabeling and duplication, leading to unreliable evaluation and limited real-world relevance. To address these limitations, this paper introduces SecVulEval, a context-aware benchmark designed to evaluate LLMs on vulnerability detection with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model’s ability to localize and understand vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for benchmarking vulnerability detection in realistic software development scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated state-of-the-art LLMs in both standalone and multi-agent settings. Results on our dataset indicate that current models remain far from accurately identifying vulnerable statements within a given function, although agent-based approaches provide modest but promising improvements. The best-performing Claude-3.7-Sonnet-driven agent achieves an F1-score of 23.83% for vulnerable statement detection. We believe this benchmark can serve as a foundation for advancing context-aware vulnerability detection with LLMs.

PRWeaver: Evaluating LLM-Based Code Auditors against Long-Horizon Malicious Pull Requests

The results show that access to repository history is insufficient: concealment becomes most effective when benign and malicious changes jointly occupy the auditor's active review context or when the stated purpose plausibly accounts for the attack-bearing diff.

Large Language Model Vulnerabilities

VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection

Recent advances in LLM-based vulnerability detection have shown promising results, while coding agents further extend this capability from isolated code snippets to complete repositories. This shift requires agents to autonomously explore repositories and locate vulnerability-relevant code, instead of performing detection on preselected functions. However, existing benchmarks primarily focus on vulnerability classification over preselected code snippets, limiting their ability to evaluate coding agents in repository-level vulnerability detection. Moreover, without fine-grained vulnerability trace annotations, the capability limitations underlying the detection process remain difficult to explore. To address these limitations, we present \textbf{VulnGym}, a real-world repository-level benchmark for evaluating vulnerability detection by coding agents. VulnGym aligns reviewed GitHub advisories with their corresponding vulnerable version repositories. It contains 184 advisories and 408 vulnerability entries across 23 repositories, with each entry annotated with line-level entry points, critical operations, and vulnerability traces. Using this fine-grained ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction. Our evaluation indicates that current coding agents remain limited in both end-to-end repository-level vulnerability detection and the construction of accurate supporting traces.