OmniVul: A Holistic, Multi-Turn Conversational Benchmark for LLM-Based Vulnerability Assessment

SecVulEval: Context-Aware Benchmarking of LLMs for Vulnerability Detection

Large Language Models (LLMs) show promise for vulnerability detection, but their evaluation is limited by the lack of high-quality benchmarks. Most existing datasets rely on coarse function-level labels, overlook fine-grained vulnerability patterns, and lack critical program context such as data/control dependencies. They also suffer from data quality issues, including mislabeling and duplication, leading to unreliable evaluation and limited real-world relevance. To address these limitations, this paper introduces SecVulEval, a context-aware benchmark designed to evaluate LLMs on vulnerability detection with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model’s ability to localize and understand vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for benchmarking vulnerability detection in realistic software development scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated state-of-the-art LLMs in both standalone and multi-agent settings. Results on our dataset indicate that current models remain far from accurately identifying vulnerable statements within a given function, although agent-based approaches provide modest but promising improvements. The best-performing Claude-3.7-Sonnet-driven agent achieves an F1-score of 23.83% for vulnerable statement detection. We believe this benchmark can serve as a foundation for advancing context-aware vulnerability detection with LLMs.

Large Language Model Vulnerabilities

Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents

VIC-RAGENT is proposed, an LLM-based multi-agent framework for effective and explainable vulnerability detection that employs a multi-stage reasoning process that progressively refines candidate vulnerabilities through preliminary inspection, reanalysis, and a final decision stage.

Multi-dimensional Vulnerability Analysis of LM Safety Guards via Role-Goal-Format Decomposition

Current safety guard models for Large Language Models (LLMs) often fail to block complex Jailbreaking attacks because they focus on the structure of a prompt rather than its true intent. To evaluate this problem, we introduce the Role-Goal-Format (RGF) framework, which analyzes how the combination of a user’s persona, their objective, and the text format can bypass safety filters. We compared two small safety guard models: Prompt Guard 2 and Llama Guard 3. The results show that even with the same overall accuracy, they have completely opposite weaknesses. Prompt Guard 2 is easily bypassed by unusual text formats. On the other hand, Llama Guard 3 lacks semantic understanding, missing 76.7% of virus-related attacks. Furthermore, both models exhibit a common vulnerability to contextual breaking; they fail to activate safety alignment when harmful objectives are framed within authoritative and benign personas, such as Teacher, Engineer, and Investigator. These findings highlight that small safety guard models have clear limitations and require multi-layered defenses that understand both text structure and actual intent.

From Jargon to Action: Evaluating Open-Weight LLMs for Plain Language Summaries of Pentest Reports

Penetration testing reports are a critical artifact in the cybersecurity workflow, yet their technical complexity frequently limits their utility for non-specialist stakeholders involved in risk remediation decisions. This paper investigates the feasibility of using four open-weight large language models, DeepSeek-r1:32b, Qwen3.5:35b, Gemma4:31b, and GLM-4.7-flash:32b, to generate plain-language summaries of penetration testing reports. A corpus of 65 publicly available pentest reports was used for evaluation. Model outputs were assessed across four dimensions: readability, technical term density, semantic similarity to the source document, and factual correctness using LLM-as-a-judge evaluation. Two classical extractive methods, LSA and TextRank, were included as baselines. Readability analysis using seven established metrics showed that Qwen3.5 and Gemma4 produced the most accessible summaries, reducing mean Flesch Reading Ease scores from 25.2 in the originals to 49.2 and 52.1 respectively, and lowering grade-level scores from post-graduate to high-school equivalents. Results across the remaining evaluation dimensions further indicate that appropriately selected open-weight LLMs can produce accessible and factually grounded summaries of technical security documents, offering a practical alternative to proprietary solutions in privacy-sensitive deployment contexts.

VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

VICBench enables robust evaluation of vulnerability detection approaches and shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort.