Skip to content

RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

Sep 2026 · 0 citations
Computer Science

TL;DR

RecLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences is introduced, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences for each paper, and what counts as a successful reproduction is fixed.

Abstract

Reproducing a machine learning paper involves most research steps, from installing software and debugging to running experiments, work that AI agents increasingly do. We introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences. For each paper we fix in advance the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget. An agent must reproduce that result using the paper and whatever its authors released. What the authors released decides the difficulty tier. Run-tier releases include code, data, and weights; Retrain-tier releases lack weights, so the agent trains the model; Reimplement-tier releases lack code, so the agent writes it. A separate language model grades runs from logs and outputs rather than agents'reports. We run four agents once per paper; the best agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement, where every agent does worst. Failed attempts use on average 29% of their budget, so most stop with budget left. The most common agent error is writing the method without checking any part against the paper's numbers, in 63 of 400 runs.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

TuiML: Machine Learning for AI Agents

TuiML is a self-contained machine-learning library built for AI agents, with native algorithms across supervised, unsupervised, time-series, data handling, tuning, and evaluation tasks, and Benchmarks show TuiML remains predictively competitive with scikit-learn and Weka.

Nilesh Verma, N. Lim, Albert Bifet et al. · 0 citations

AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?

Recent gains in language model capability have come more from data than from architecture. Frontier labs and data companies produce verifiable agentic tasks, which supervised finetuning and reinforcement learning then turn into capability.This production line still rests on human labour and on human-in-the-loop collabo...

Haotian Luo, Hao-Yu Wang, Ze-Yu Qin et al. · 0 citations
Preprint Aug 2026

AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement

This work presents AI4AI\mbox{-}Bench, 10 frozen research repositories spanning 10 training algorithm families, and releases the task suite, the evaluators and every scored submission, so that the measurement can be repeated as these systems change.

Yi-Zhe Chi, Wen-Yi Li, De-Yao Hong et al. · 11 citations · ⚡3
Preprint Aug 2026

TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development

Auto-research agents now run machine-learning development unattended for hours, revising data pipelines, models, and validation from their own feedback, yet on most competitions they still finish below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final sub...

J. Yan, Weiwei Sun, Si-Jie Li et al. · 0 citations

Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

This work introduces AgentActionBench, a process-oriented benchmark for evaluating agent-based experiment reproduction across ML and AI4Science domains, and uses an MCP-based Action Recorder to capture agents' behaviour throughout the reproduction process and evaluates the resulting traces with paper-specific rubrics.

Han-Hua Hong, Yi-Zhi Li, H. Luu et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.