Skip to content

SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

Sep 2026 · 0 citations · 58 references
Computer Science

TL;DR

SecProbe is introduced, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks and achieves comparable agent ability estimates while requiring agents to solve up to 29.5\% fewer tasks.

Abstract

Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale. We introduce SecProbe, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks. From observed performance, \textsc{SecProbe} estimates agent ability and identifies where additional evidence is most informative, selecting existing tasks or synthesizing new ones accordingly. As one use case, we construct 353 tasks spanning six programming languages and 151 CWE types and evaluate nine frontier models with two agent harnesses. Success rates peak at 28.33\%, highlighting substantial gaps in vulnerability recognition and repair. Compared with random and one-shot baselines, \textsc{SecProbe} achieves comparable agent ability estimates while requiring agents to solve up to 29.5\% fewer tasks. These results support adaptive evaluation as an efficient and discriminative approach to assessing cybersecurity vulnerability awareness.

View source

Similar papers

Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

VLoc Benchmark results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.

Aman Priyanshu, Supriti Vijay, Kimia Majd et al. · 2 citations · ⚡1
#artificial intelligence Preprint Oct 2026

ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding

As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing work evaluates related agent behaviors from separate perspectives, but lacks a systematic framework for unifying these behaviors. To bridge this gap, we introduce Para...

Han-Jun Luo, Xiu-Cheng Zhang, Zhuo-Ning Xu et al. · 0 citations
#small language model Preprint Sep 2026

PrivEscalate: Measuring and Augmenting the Threat of LLM-Automated Linux Privilege Escalation

PrivEscalate is presented, a large-scale benchmark for Linux privilege escalation, comprising 531 Dockerized scenarios spanning 14 sub-categories and PrivEscalate, a domain-specialized wrapper that augments a generic ReAct agent with deterministic enumeration, category matching, and step planning that improves over pri...

Yi-Xuan Liu, Zi-Long Zhen, Yin Wu et al. · 0 citations
Review Open access Sep 2026

Toward Dynamic and Risk-Aware Evaluation of Cybersecurity LLMs: A Survey and the RIRAG Framework

This survey analyzes recent LLM cybersecurity benchmarks through three evaluation paradigms: knowledge-oriented, task-oriented, and holistic evaluation, and introduces the Reflective and Iterative Retrieval-Augmented Generation framework, a dynamic and risk-aware evaluation architecture for cybersecurity LLMs.

Ravi Prasad, Feroz Ahmed, Shohel Rana et al. · 0 citations
Book Open access Oct 2026

To Think or Not to Think: Evaluating LLM Reasoning and Agents in Vulnerability Detection

Large language models (LLMs) are increasingly studied for software vulnerability detection, but the benefits of recent reasoning-enabled (“thinking”) modes remain unclear. At the same time, agentic LLM designs aim to enhance analysis through structured collaboration, while practical software security workflows require...

Hua-Beng Tan, Ratnadira Widyasari, Merve Astekin et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.