SecProbe is introduced, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks and achieves comparable agent ability estimates while requiring agents to solve up to 29.5\% fewer tasks.
Abstract
Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale. We introduce SecProbe, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks. From observed performance, \textsc{SecProbe} estimates agent ability and identifies where additional evidence is most informative, selecting existing tasks or synthesizing new ones accordingly. As one use case, we construct 353 tasks spanning six programming languages and 151 CWE types and evaluate nine frontier models with two agent harnesses. Success rates peak at 28.33\%, highlighting substantial gaps in vulnerability recognition and repair. Compared with random and one-shot baselines, \textsc{SecProbe} achieves comparable agent ability estimates while requiring agents to solve up to 29.5\% fewer tasks. These results support adaptive evaluation as an efficient and discriminative approach to assessing cybersecurity vulnerability awareness.
VLoc Benchmark results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.
Aman Priyanshu, Supriti Vijay, Kimia Majd et al.· arXiv.org· 2 citations· ⚡1
As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing work evaluates related agent behaviors from separate perspectives, but lacks a systematic framework for unifying these behaviors. To bridge this gap, we introduce Para...
Han-Jun Luo, Xiu-Cheng Zhang, Zhuo-Ning Xu et al.· 0 citations
PrivEscalate is presented, a large-scale benchmark for Linux privilege escalation, comprising 531 Dockerized scenarios spanning 14 sub-categories and PrivEscalate, a domain-specialized wrapper that augments a generic ReAct agent with deterministic enumeration, category matching, and step planning that improves over pri...
Yi-Xuan Liu, Zi-Long Zhen, Yin Wu et al.· 0 citations
This survey analyzes recent LLM cybersecurity benchmarks through three evaluation paradigms: knowledge-oriented, task-oriented, and holistic evaluation, and introduces the Reflective and Iterative Retrieval-Augmented Generation framework, a dynamic and risk-aware evaluation architecture for cybersecurity LLMs.
Ravi Prasad, Feroz Ahmed, Shohel Rana et al.· Journal of Cybersecurity, Di...· 0 citations
The findings show that LLMs can approximate structured cybersecurity reasoning under controlled representations, but do not apply it robustly, which has important implications for the design and evaluation of AI-assisted security decision-support systems.
Large language models (LLMs) are increasingly studied for software vulnerability detection, but the benefits of recent reasoning-enabled (“thinking”) modes remain unclear. At the same time, agentic LLM designs aim to enhance analysis through structured collaboration, while practical software security workflows require...
Hua-Beng Tan, Ratnadira Widyasari, Merve Astekin et al.· Proceedings of the 41st IEEE...· 0 citations