Skip to content

SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories

Sep 2026 · arXiv.org · Vol abs/2609.29465 · 1 citation · 62 references
Computer Science

TL;DR

SWE-Prometheus is presented, a benchmark for the broader task of improving repository engineering governance, showing why repository-governance evaluation should report improvement, behavior preservation, evidence quality, and coverage together.

Abstract

Large language model based coding agents have made substantial progress on repository-level software engineering tasks. Existing repository benchmarks, however, usually start from a human-identified issue and evaluate whether a patch satisfies a functional signal. We present SWE-Prometheus, a benchmark for the broader task of improving repository engineering governance. Each task provides a fixed snapshot and an open-ended objective, requiring the agent to identify risks, prioritize interventions, and verify the resulting changes. SWE-Prometheus evaluates six governance dimensions through paired evidence, clean-environment probes, behavior gates, and two independent teacher ratings of the same evidence. The benchmark contains 60 repositories; ten models are evaluated on a shared 22-repository public subset, where mean Normalized Governance Improvement ranges from 0.0568 to 0.5760 and observed behavior-breakage rates range from 0% to 23%. On a frozen ten-repository batch, a repository-blind template obtains mean NGI 0.272, but its gains concentrate in Tests&CI, Quality Gates, and Documentation; it improves Reproducible Environment and Dependency&Security on none of the repositories. This baseline makes the distinction between adding governance artifacts and producing execution-backed improvements measurable. The no-op condition has median NGI zero and standard deviation 0.073; two teachers agree exactly on 57 of 60 dimension scores for the same no-op evidence. For the two highest conditional-mean systems, common-valid NGI is similar, while full-pool comparisons that include behavior failures favor Kimi-K3. These results show why repository-governance evaluation should report improvement, behavior preservation, evidence quality, and coverage together.

View source

Similar papers

Preprint Sep 2026

Methodological Harness in Agentic Software Engineering: An Empirical Study on Mining Software Repositories

Context. Agentic software engineering requires mechanisms to coordinate and govern agents'work. The framework motivating this study proposes a methodological harness with eight mechanisms: context engineering, persistent shared knowledge, executable specifications, N-version mindset and parallel agents, normative speci...

J. Díaz, Jorge Pérez, Sergio Gil-Borrás · 0 citations
#artificial intelligence Preprint Oct 2026

Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents

The evaluation of four LLMs under two agent scaffolds shows that modern agents suffer from coding compliance issues: although agents produce functionally correct patches, they still violate 43.1 percent of applicable project policies, with nearly half of all violations occurring during intermediate execution steps.

Hải Đăng Trương, Rayner Goh, Thanh Le-Cong et al. · 0 citations
Review Sep 2026

WideSWE: Can Coding Agents Coordinate Changes Across Repositories?

Coding-agent evaluation has progressed from resolving individual issues to carrying out long-horizon development, yet task completion is still largely assessed within a single codebase. In software ecosystems, many features and bug fixes require coordinated changes across multiple repositories. We introduce WideSWE to...

Bao-Yi Wang, Xing-Liang Wang, Jin-Yang Wu et al. · 0 citations
Review Sep 2026

Spotlights: Discovering Improvement Opportunities in Software Repositories

Spotlights is presented, a system that performs optimization-opportunity discovery as a distinct and empirically evaluable step between a broad engineering objective and subsequent implementation and validation.

U. Barzelay, Ophir Azulai, Idan Friedman et al. · 0 citations

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

E2E-SWE is introduced, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end and finds substantial variation in end-to-end repository generation ability, providing strong model differentiation while leaving considerable headroom for future progress.

Hantian Ding, Chloe Bi, Jia-Cheng Zhu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.