Skip to content

WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

Sep 2026 · arXiv.org · Vol abs/2609.27490 · 1 citation · 55 references
Computer Science

TL;DR

WhatWorkedBench links acquisition, inference, program structure, and delivery through a delivered response surface of configuration scores, which evaluates this experimental understanding through a delivered response surface of configuration scores.

Abstract

AI research agents must predict the effects of computational changes after budgeted experiments. WhatWorkedBench evaluates this experimental understanding through a delivered response surface of configuration scores. Agents inspect workflow code and buy measurements; exhaustive CPU references score conditional component effects, mean pair interactions, configuration choice, and delivery. The catalog spans 36 task conditions, 30 sources, 8 workflow families, and 1248 indexed configuration records, with 4,206 numerical controls. At eight purchased measurements plus two free anchors, pair-effect ridge selects an exact optimum on 15 of 22 sources; 13 of these cases have at least one conditional-effect error exceeding 10% of the task utility range. Shared-estimator comparisons measure acquisition and reconstruction on common observations. In a prospective typed study on 12 four-factor sources, DeepSeek Flash submits 12/12 direct tables and gains 0.147 recovery over a Gaussian process (GP) fitted to the same observations. Pro delivers 11/12 artifacts, with an all-attempt GP difference of -0.001 and a delivered-only difference of +0.063. On six prespecified new agent-evaluation sources, Flash and Pro gains are 0.149 and 0.074. In eight typed six-factor episodes, seven final tables satisfy verified code equivalences. Four fresh agents pass all six registered rules through named estimators and deliver consistent tables at 0.682 recovery versus 0.710 for separate direct-table runs. WhatWorkedBench links acquisition, inference, program structure, and delivery.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

AgentIdeaBench is introduced, a multidisciplinary benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration, and Scientific World Modeling is explored, a generation-time loop that refines a draft hypothesis through structured thought experiments.

Yunxiang Mo, Tianshi ZHENG, Yi-Sen Gao et al. · 1 citation
#artificial intelligence Preprint Aug 2026

Science sandboxes measure the scientific capability of AI agents

This work instantiates science sandboxes, a framework for studying this capability in AI agents through repeated cycles of experimentation, feedback, and hypothesis revision, in two biological settings, models of regulatory genomics and protein fitness prediction, and examines the capabilities of frontier agents.

Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai et al. · 1 citation
#artificial intelligence Preprint Sep 2026

DeltaSelect: Affordable A/B Testing for Coding Agents

The paper presents DeltaSelect, an open-source method that identifies tasks whose one-run results consistently track full-benchmark performance using Pearson correlation, maps fractional verifier results to a common score using linear regression, and selects a fixed task set within a dollar budget.

Nicholas J. Conn · 0 citations
#natural language process... Preprint Sep 2026

EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

This work introduces early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task within each task, and instantiates EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features and hal...

Yu-Ling Shi, Zhensu Sun, Jun-Sen Dong et al. · 4 citations · ⚡1
#artificial intelligence Preprint Aug 2026

K-Bench: measuring model performance on real scientific agent requests

K-Bench 01, an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web and run end to end by nine frontier models in identical sandboxes, yielding 1,602 completed agent runs is reported, arguing that the informative quantity for scientific agents is not a leaderboard position but the joi...

Aubrey M. Brueckner, Darshil Patel, Yu-Huan He et al. · 2 citations
#artificial intelligence Preprint Sep 2026

Substrate-Aware AI Agents: Execution Context as a First-Class Input

A minimal execution contract induces proactive structural adaptation in generated programs, shifting computation away from unconstrained allocations and substantially improving observed resource-time profiles before execution, establishing a controlled proof of concept for substrate-aware agent planning.

Manu Agrawal · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.