LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review...
A benchmark for predicting final output and checkpoint state from source and input alone, extends CRUXEval-style output prediction with paired shorter- and longer-trace inputs and checkpoints inside and after a loop and exposes errors hidden by short-output scores alone.
Xiao-Hong Chen, David Bucur, Cheng-Long Ma et al.· 0 citations
Reproduction test generation translates a natural-language issue description into executable tests that fail on the original code and pass after the issue is resolved, providing executable evidence for verifying candidate patches. Existing benchmarks are constructed for individual programming languages, preventing a un...
Kazuki Kusama, Sota Nakashima, Haruka Tokumasu et al.· 0 citations
Coding agents need to establish that a program satisfies a specification and that the specification captures the requested behavior. We study how expert diagnosis of verification failures can become reusable guidance for this work. Our approach combines executable language definitions in the K framework with a kit of p...