Skip to content

EAIRA: Establishing a methodology for evaluating LLMs as scientific research assistants

Sep 2026 · The international journal of high performance computing applications · 0 citations · 38 references

TL;DR

A multifaceted methodology for Evaluating AI models as scientific Research Assistants (EAIRA) developed at Argonne National Laboratory is described, designed to be generalizable to a wide range of scientific domains.

Abstract

Recent advancements have positioned AI, and particularly Large Language Models (LLMs) as transformative tools for scientific research, capable of addressing complex tasks that require reasoning, problem-solving, and decision-making. Their exceptional capabilities suggest their potential as scientific research assistants, but also highlight the need for holistic, rigorous, and domain-specific evaluation to assess effectiveness in real-world scientific applications. This paper describes a multifaceted methodology for Evaluating AI models as scientific Research Assistants (EAIRA) developed at Argonne National Laboratory. This methodology incorporates four primary classes of evaluations. (1) Multiple Choice Questions to assess factual recall; (2) Open Response to evaluate advanced reasoning and problem-solving skills; (3) Lab-Style Experiments involving detailed analysis of capabilities as research assistants in controlled environments; and (4) Field-Style Experiments to capture researcher-LLM interactions at scale in a wide range of scientific domains and applications. These complementary methods enable a comprehensive analysis of LLM strengths and weaknesses with respect to their scientific knowledge, reasoning abilities, and adaptability. Recognizing the rapid pace of LLM advancements, we designed the methodology to evolve and adapt so as to ensure its continued relevance and applicability. This paper describes the methodology’s state at the end of February 2025. Although developed within a subset of scientific domains, the methodology is designed to be generalizable to a wide range of scientific domains.

View source

Similar papers

Open access Aug 2026

Remaking the landscape of scientific expertise: The emergent research program of total automation in science

This article examines an emergent research program aimed at the complete automation of science, analyzing how various automation projects funded by DARPA and others are working to replicate traditionally human scientific competencies. Rather than directly attempting to create fully autonomous robot scientists, these pr...

David Peterson, Bernard J. Koch, Ramya Natarajan et al. · 0 citations
#generative ai Review Open access Sep 2026

Artificial intelligence in scientific research: a practical guide to tools and ethical considerations

Artificial intelligence (AI), including generative AI and large language models (LLMs), is rapidly transforming scientific research. It is increasingly being integrated into multiple stages of the research process. The public release of ChatGPT in November 2022 contributed to increased adoption of these technologies, c...

E. M. Alakraa, Amgad Abd El Raouf Farhat · 0 citations
#generative ai Review Open access Sep 2026

Beyond Automation: How Generative AI is Reshaping Research Evaluation and Scholarly Communication

A comprehensive mixed-methods investigation into the impact of GenAI on research evaluation and scholarly communication is presented, combining a large-scale survey of 2,850 researchers, editors, and graduate students with a quantitative analysis of 18,000 manuscripts submitted to 12 peer-reviewed journals between 2022...

Ravi Ranjan, Ritesh Kandari, Anirudh Gupta, Divya · 0 citations
#large language models Open access Oct 2026

A multifactor model for assessing programming knowledge based on the synchronization of AI analytics and expert evaluation

The widespread adoption of generative AI tools has created new methodological challenges for the validity of assessment in higher education. If a neural network can write code, how can a student's actual knowledge be assessed objectively? This study proposes a multifactor AI-assisted assessment model for the fair e...

Z. Nurbekova, M. Kemelbek, A. Sagimbayeva et al. · 0 citations
Open access Oct 2026

Expanding the role of large language models in biology assessment

T he rapid development of large language models, or LLMs, has created new opportunities to reconsider how instructors approach time-intensive tasks, such as grading and feedback. Naidu et al. provide a timely examination of this application in undergraduate biology education through their evaluation of LLM-based gradin...

Jordan A. Smink · 0 citations

LLJ Cards: Best practices for the Use of LLMs as Judges

LLJ Cards is introduced, a framework that synthesizes best practices from measurement theory, natural language generation, and machine learning literature into practical guidelines for LLJ-based evaluations and provides a structured framework for applying these principles in the design and reporting of automated evalua...

Khaoula Chehbouni, Melina Medjdoub, Florian Carichon et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.