VICBench enables robust evaluation of vulnerability detection approaches and shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort.
Abstract
Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.
Large Language Models (LLMs) show promise for vulnerability detection, but their evaluation is limited by the lack of high-quality benchmarks. Most existing datasets rely on coarse function-level labels, overlook fine-grained vulnerability patterns, and lack critical program context such as data/control dependencies. They also suffer from data quality issues, including mislabeling and duplication, leading to unreliable evaluation and limited real-world relevance. To address these limitations, this paper introduces SecVulEval, a context-aware benchmark designed to evaluate LLMs on vulnerability detection with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model’s ability to localize and understand vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for benchmarking vulnerability detection in realistic software development scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated state-of-the-art LLMs in both standalone and multi-agent settings. Results on our dataset indicate that current models remain far from accurately identifying vulnerable statements within a given function, although agent-based approaches provide modest but promising improvements. The best-performing Claude-3.7-Sonnet-driven agent achieves an F1-score of 23.83% for vulnerable statement detection. We believe this benchmark can serve as a foundation for advancing context-aware vulnerability detection with LLMs.
Md Basim Uddin Ahmed, Nima Shiri Harzevili, Jiho Shin et al.· AIware· 2 citations
Recent advances in LLM-based vulnerability detection have shown promising results, while coding agents further extend this capability from isolated code snippets to complete repositories. This shift requires agents to autonomously explore repositories and locate vulnerability-relevant code, instead of performing detection on preselected functions. However, existing benchmarks primarily focus on vulnerability classification over preselected code snippets, limiting their ability to evaluate coding agents in repository-level vulnerability detection. Moreover, without fine-grained vulnerability trace annotations, the capability limitations underlying the detection process remain difficult to explore. To address these limitations, we present \textbf{VulnGym}, a real-world repository-level benchmark for evaluating vulnerability detection by coding agents. VulnGym aligns reviewed GitHub advisories with their corresponding vulnerable version repositories. It contains 184 advisories and 408 vulnerability entries across 23 repositories, with each entry annotated with line-level entry points, critical operations, and vulnerability traces. Using this fine-grained ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction. Our evaluation indicates that current coding agents remain limited in both end-to-end repository-level vulnerability detection and the construction of accurate supporting traces.
Kexing Ji, Jiachen Liu, Enze Hu et al.· 0 citations
Smart contract vulnerability detection requires evaluation protocols that separate real representation signal from dataset-specific artifacts. DIVE provides lifecycle-based tabular features for Ethereum smart contracts, but benchmark performance alone cannot show whether a dominant feature group is useful or only benefits from having many columns. This study examines Opcode Distribution features using 22,330 contracts, 397 processed features, and eight DASP-aligned vulnerability labels. Five multi-label learning configurations were evaluated under 3 x 5 repeated cross-validation, followed by global feature-group ablation, size-controlled random opcode ablation, per-label degradation analysis, cumulative stability analysis, and opcode-profile group-aware robustness checking. MultiOutput LightGBM achieved the best baseline performance, with Micro-F1 of 0.91396, Macro-F1 of 0.82464, and Macro-PR-AUC of 0.90146. Removing the full Opcode Distribution group reduced Macro-F1 to 0.78745, while removing a same-sized random opcode subset produced Macro-F1 of 0.82404. The findings indicate that Opcode Distribution acts as a collective predictive representation rather than a feature-count artifact, without implying causal vulnerability mechanisms.
Astrid Pranadani, Dhani Ariatmanto· Data Science Journal of Comp...· 0 citations
Model Context Protocol (MCP) servers have become essential for connecting large language models (LLMs) with external tools, data sources, and execution environments. However, their privileged access and bridging role introduce significant security risks. Despite many LLM-based detection methods, it remains unclear whether they can semantically differentiate paired vulnerable and fixed MCP servers rather than just identifying risky API calls. In this work, we present a dedicated benchmark for pairing MCP server vulnerability detection, comprising 75 paired vulnerable and fixed MCP tools synthesized by Gemini-1.5-Pro and manually cross-validated. We adopt the Matched Pair Accuracy (MPA) metric to evaluate detection methods. Claude Haiku-4.5 achieves the best MPA of 0.853, while GPT-5-mini attains 0.842 F1 but only 0.653 MPA, showing that conventional metrics can overestimate paired detection performance. Our benchmark provides a new perspective for systematic, reproducible, and MCP-specific security evaluation.
The Security Mutation Score (SMS) is proposed, a metric that classifies mutant kills into semantic, functional, incidental, and crash categories using operator-aware heuristics, distinguishing genuine security awareness from coincidental detection.
VIC-RAGENT is proposed, an LLM-based multi-agent framework for effective and explainable vulnerability detection that employs a multi-stage reasoning process that progressively refines candidate vulnerabilities through preliminary inspection, reanalysis, and a final decision stage.
Liyou Chen, Hailong Sun, Xiang Gao et al.· 0 citations