When code does not run: reproducibility challenges in materials machine learning benchmarks

Abstract

Reproducible benchmarks are essential for advancing data-driven materials research by standardizing datasets, prediction tasks, and evaluation protocols. In machine learning benchmarks, reproducibility is often assumed to follow from releasing source code and documented software dependencies. However, dependency specifications must be sufficiently complete to reconstruct an executable software environment from a defined, isolated starting point. Because modern package ecosystems are complex and evolve over time, automated environment reconstruction and sandboxed executability checks are necessary prerequisites for assessing numerical reproducibility. Here, we systematically evaluate this aspect of reproducibility using MatBench, a benchmark platform for materials property prediction. Using the original dependency metadata without modification, only 2 of 28 submissions produced environments that could be installed and pass import checks. Rule-based and human-assisted remediation increased this number to 26 of 28. Among these, 13 submissions successfully completed a selected representative benchmark task and generated outputs, corresponding to 46.4% of all evaluated submissions. For these successfully re-executed submissions, the regenerated five-fold mean root mean square error (RMSE) values were close to the originally reported results. Similar executability issues were also observed on the JARVIS-Leaderboard. These findings demonstrate that reconstructing runnable environments from currently provided benchmark metadata remains challenging. We identify common causes of failure and propose platform-level practices for improving the long-term reproducibility and executability of machine learning benchmarks in materials informatics.