Building a Self-Improving Agentic Harness for RSE Projects

Abstract

AI coding agents have moved from curiosities to capable collaborators in only a few years [4,5]. Early evidence was mixed [2,5], and as late as 2025 studies showed limited success on real software tasks that varied by domain and developer experience [1]. Over the period we report on (November 2025 – August 2026), both models and practice improved: orchestrator-driven multi-agent workflows [19], repeated adversarial review [6,7], context engineering, MCP tool calling, and project-specific skills (stored, version-controlled prompts invoked by name) [12,13]. Practice, discipline, and tooling separate agentic engineering from "vibe coding" [18]; adoption ranges from prohibition to treating the harness as shared infrastructure [3], and industry teams now build "software factories" [20,21,22]. This poster presents a semi-autonomous, self-improving, spec-driven [10,11], end-to-end development lifecycle, agentic software factory initially prototyped within the HyperShell project [14,15,16]; composed of eight agent skills plus wiki as context. Each feature moves through a fixed sequence (shape, plan, build, blind adversarial review, publish) and leaves a dated design record in the repository. We then ported the factory to a second research code, xdu, a parallel file-metadata indexer for HPC file systems, to test whether the method transfers. As of September 2026 the two projects hold 23 feature records and 60 recorded harness decisions. Research software exists to support claims that other people must believe, so the question for RSEs is where a human must stay in the loop to validate agent output. The factory places the human at four gates: shaping the goal and its explicit exclusions [8,9]; resolving any ambiguity the agent marks rather than guesses; triaging review findings the blind reviewer could not reproduce with an executed command; and signing off whenever a confirmed finding touches a high-blast-radius file. Between gates the agents work unattended, and correctness rests on tests and executed evidence. The harness also improves itself. Every lifecycle skill logs friction in the harness at near-zero cost, and a separate, human-gated maintenance skill applies fixes under guardrails that forbid quietly weakening a gate, with a ledger that stops rejected changes from returning [7]. The poster includes the complete bill of materials for the harness, the skills and templates that drive it [15,16], one feature traced end to end, and examples of where humans intervened at each gate. Our goal is to give RSEs a reusable method and to open a discussion of where oversight belongs [17].