An overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis, with a specific emphasis on scenarios in which the models are directly integrated with experimental tools.
Abstract
This work provides an overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis. After providing a brief overview of the key technologies behind the current generation of AI agents based on LLMs, we summarize the different approaches to evaluating these models in the context of materials science and in particular on materials synthesis, with a specific emphasis on scenarios in which the models are directly integrated with experimental tools. We discuss evaluation strategies spanning knowledge and reasoning benchmarks, tool-use benchmarks, and closed loop benchmarks involving the interaction with experimental systems or realistic virtual tools. We use atomic layer deposition (ALD) as a case study, emphasizing how existing approaches in the literature both build from general approaches used beyond materials science and can be generalized to other materials synthesis techniques. Finally, we provide a practical evaluation framework to evaluate LLMs in the context of materials synthesis
In a single campaign of 18 autonomous experiments using LiCoO2 (001) thin-film deposition as a testbed, SynAgent synthesized highly crystalline films and evolved an understanding of how the substrate temperature governs crystallization, discovering an abrupt threshold and a narrow optimal growth window at 650-690 {\deg...
Izumi Takahara, K. Nishio, Akira Aiba et al.· 1 citation
CompMat-Bench is introduced, a benchmark of 94 tasks derived from recently published computational materials studies, each asking agents to complete a step toward achieving the study's scientific goal, and provides a basis for comparing agents on the steps of real materials research and for analyzing agent failure mode...
Chen-Mu Zhang, Levi C. Felix, Jun-Jie Zhang et al.· 0 citations
In this review, recent advances in material‐science agents from the complementary perspectives of autonomous design and autonomous experimentation are highlighted and a staged roadmap is proposed to facilitate the development of a standardized, reliable, and increasingly autonomous ecosystem for next‑generation materia...
Tong-Bo Jiang, Zheng-Yang Zhang, Ming-Shuo Nie et al.· Advanced Functional Material...· 0 citations
MolDesignBench is proposed, a scenario-grounded benchmark that more closely reflects real-world molecular design for evaluating tool-augmented LLM agents and establishes MolDesignBench as a rigorous testbed to guide future research on chemical agents.
Yongjun Jeong, Hanbum Ko, Yejun Kim et al.· 0 citations
Tool-using agents increasingly require a common substrate for building, training, evaluating, distilling, and maintaining agentic systems, yet existing tooling typically addresses these stages in isolation. We present AgentTune, a source-available Python library that provides a unified framework for developing and opti...
R E Zera Marveen Lyngkhoi, Abhivansh Gupta, Vidushee Vats et al.· 0 citations
Low-level hierarchical discussion in HiMatGen provides an approach to materials design that sustains the exchange of scientific knowledge and computational evidence throughout iterative exploration as the agent population grows.