Skip to content
Log in
Preprint

One Recipe, Many Harnesses: What Self-Evolution Encodes Across Languages and Models

Siqi Yang Qianlan Yang Yu-Xiong Wang Saurabh Pujar Martin Hirzel
Aug 2026 · 2 citations · 36 references
Computer Science

TL;DR

Together, these results recast the evolved harness as a legible compensation layer, shaped jointly by the language's engineering demands and the model's behavioral gaps, rather than an opaque benchmark-tuned scaffold.

Abstract

Self-evolving harnesses are closed-loop systems in which an agent inspects its own rollouts and edits its prompts, tools, and memory. They reliably improve coding agents in evaluations, but prior work reports aggregate gains rather than analyzing what the evolved artifacts encode. It therefore remains unclear whether they encode benchmark-specific adaptations, language-specific engineering knowledge, or compensation for limitations of the underlying model. We disentangle these factors by holding an evolution recipe fixed across a grid of eight programming languages (Multi-SWE-Bench) and three base models, and analyzing the resulting harnesses. The recipe routes every edit through a typed failure signal and records it as a falsifiable contract, making each modification attributable after evolution. Four findings emerge. (1)The loop improves held-out solve rates over both a minimal seed and the manually designed mini-SWE-agent scaffold in most cells, but with two null regions. (2)Gains compensate recoverable execution defects, where defect mass is near zero, and gain is near zero; which defect dominates is cell-specific. A harness closes the gap between what a policy can do and what it does. (3)Evolved harnesses share an abstract playbook across languages but instantiate it with almost disjoint language ecosystem machinery. (4)The shared core transfers and can be distilled into one universal harness, while an ecosystem margin resists both and requires native re-evolution. Together, these results recast the evolved harness as a legible compensation layer, shaped jointly by the language's engineering demands and the model's behavioral gaps, rather than an opaque benchmark-tuned scaffold.

arXiv Semantic Scholar
View source

Similar papers

Preprint Jul 2026

DarwinX: Evolving Agent Harnesses Through Natural Selection

DarwinX is introduced, which treats self-evolution as selection over a population of harnesses with the model frozen: a preserve-and-extend contract admits only variants that extend coverage without regressing, an archive keeps alternative lineages for recombination, and failure-, teacher-, and self-derived evidence share one edit interface.

Yifang Zhang, Yutong Dai, Juntao Tan et al. · 1 citation
Preprint Jul 2026

Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the target behavior. This is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, while modification requests describe what the system should do and repositories are organized by files and modules. Code search, repository indexing, and long-context processing ease inspection, but still leave this behavior-to-code mapping to be recovered by hand. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase via static analysis and LLM-assisted structuring, linking each behavior to its corresponding source. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides agents from high-level behaviors to relevant implementation details and verifies candidate locations against the current source. On diverse modification requests from two open-source harnesses, Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens, with the largest gains on scattered sites, rarely executed paths, and cross-module interactions. Evolving complex agentic systems thus depends not only on generating edits, but also on determining where those edits should be made.

Ruhan Wang, Yucheng Shi, Zongxia Li et al. · 7 citations
Preprint Aug 2026

HELIX: Model-Harness Co-evolution for Recursive Self-Improvement

HELIX provides an auditable interface for studying model-harness co-evolution for recursive self-improvement and expands current capability and creates learning signal for the next model; model updates motivate the next round of harness evolution.

Tianyu Fan, Chao Huang · 3 citations · ⚡1
Preprint Jul 2026

Recursive Harness Self-Improvement

Recursive Harness Self-Improvement is introduced, which represents the harness as a prompt-level specification of the agent loop and iteratively refines it using pairwise feedback over its own revision history, suggesting RHI as a practical algorithm for continual learning within the paradigm of model--harness co-evolution.

Hyunin Lee, Jinglue Xu, Jeffrey Seely et al. · 9 citations
#artificial intelligence Preprint Aug 2026

A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations

This work introduces a random variant sampler that applies common semantics-preserving transformations (SPTs) - spanning control-flow rewrites, dead-code injection, and identifier renaming - to produce perturbed variants, demonstrating that even top frontier models are susceptible to semantics-preserving perturbations.

Hasan Mahmud, Shreya Gupta, Isha Chaudhary et al. · 1 citation