Skip to content
Preprint

Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

Oct 2026 · 0 citations
Computer Science

TL;DR

Agent4Sci is introduced, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains and gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems.

Abstract

Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).

View source

Similar papers

#artificial intelligence Review Open access Sep 2026

ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

The introduction of ScientistTwo, a fully autonomous multi-agent framework designed to realize problem-driven autonomous research, and its results show that ScientistTwo is not merely an assistive tool but an autonomous scientific pioneer capable of pushing the frontiers of human discovery.

Jaehyun Nam, Jinsung Yoon, Yan Pan et al. · 0 citations

How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orc...

Kirill Brilliantov, Alejandro Hernández-Cano, Emmanuel Abbé · 0 citations

WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

WhatWorkedBench links acquisition, inference, program structure, and delivery through a delivered response surface of configuration scores, which evaluates this experimental understanding through a delivered response surface of configuration scores.

Jing-Jie Ning, Xue-Qi Li, Yi-Bo Kong et al. · 1 citation
Preprint Oct 2026

At the Limits of Computability: Steps Towards Autonomous AI Discovery on the Busy Beaver Frontier

Recent breakthroughs in AI for mathematics---ranging from Olympiad medal performances to resolving long-standing open conjectures---have rapidly expanded what automated systems can achieve. Yet these successes primarily concern predefined, human-formulated problems, leaving open how AI systems can autonomously uncover...

Marco Dos Santos, Vivek Veeriah, Alex Havrilla et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate...

Wen-Yu Du, Stephen Chung · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.