Automated Vulnerability Repair (AVR) has advanced rapidly across program analysis, machine learning, and Large Language Models (LLMs), but a verifiable, head-to-head comparison of AVR approaches on Python is still missing. Python underpins critical web, data, and machine-learning infrastructure, yet existing Python benchmarks accept a patch on the strength of a proof-of-concept exploit alone, or apply a functional test only on the subset of entries whose upstream project happens to ship one. Both therefore miss functional regressions, in which a patch defeats the exploit but breaks unrelated behavior. We present Vul4Py, a Python AVR benchmark in which every entry carries a paired oracle: an exploit oracle that must fail on the vulnerable revision and pass on the fixed one, together with a project-native pytest functional oracle that must pass on both. Vul4Py comprises 100 real vulnerabilities from 60 open-source projects, spanning 60 distinct CWEs and the years 2017 to 2025, each packaged with a pinned, reproducible per-instance environment. Using Vul4Py, we compare six approaches in three categories: a specialized vulnerability repair tool, directly prompted LLMs, and software engineering agents. The agents dominate: OpenHands repairs 41 of 100 vulnerabilities, against 4 for the strongest directly prompted LLM and 2 for the specialized tool, despite all three sharing the same backbone model. The paired oracle is what makes these counts trustworthy: it rejects 15 of the 119 patches that an exploit-only oracle would accept, and 98 of the 104 patches it admits are manually confirmed to be semantically equivalent to the developer's patches
Bui Tan, Ting Zhang, Ferdian Thung et al.· 0 citations
Coding agents are ranked almost entirely by resolve rate: whether their final patch passes the target tests. Yet two agents can reach the same outcome through very different processes, and a single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens. This process evidence lives in the trajectory, which records a run's searches, reads, edits, tool calls, validation, and reversions. However, raw traces are heterogeneous and hard to compare across runs. We present TraceProbe, a trajectory-diagnostic framework that recovers what resolve rate hides. TraceProbe normalizes each raw run into a canonical nine-type action taxonomy with deterministic effect labels, then applies two rule-based modules: Insight names single-trajectory anti-patterns adapted from established debugging practice (e.g., search loops, verification skips), while Converge aligns pairs of runs and classifies where their behavior diverges under controlled references. Applying TraceProbe to 2,500 trajectories from five production settings on SWE-Bench Verified, we find that (i) file choice is too coarse to separate success from failure, whereas function selection and completion behavior localize it; (ii) Insight anti-patterns act mainly as corpus-level difficulty clues, with search loops the most stable; and (iii) even resolved runs differ in how quickly they reach relevant code and how much failed work they incur. Trajectory structure thus adds auditable diagnostic context to outcomes by localizing inspection targets, suggesting failure hypotheses, and prioritizing runs for review.
Rui Shu, Chun Yong Chong, Xin Zhou et al.· 1 citation
SWE-RPG is introduced, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for Requirement Clarification and Implementation Planning, and suggests implicit-requirement recovery as a key candidate direction for improving coding agents.