Generative AI in Peer Review: An Evaluation of Capabilities, Limitations, and Responsible Implementation Strategies
TL;DR
The results demonstrate clear competency boundaries for GenAI within the peer review, particularly concerning tasks requiring nuanced professional judgment, particularly concerning tasks requiring nuanced professional judgment.
Abstract
Generative artificial intelligence (GenAI) has been increasingly integrated in academic publishing process. This study aims to access the performance of GenAI in peer review using multi-disciplinary manuscript samples. Five GenAI models underwent peer review tests, systematically evaluated for completeness, accuracy, and rationality of responses using combined manual scoring and statistical analysis. Findings indicated high accuracy in summarizing manuscript content and identifying suitable journals. However, GenAI models exhibited significant limitations in identifying scientific errors and evaluating paper innovation. The results demonstrate clear competency boundaries for GenAI within the peer review, particularly concerning tasks requiring nuanced professional judgment. This study clarifies the potential application scope of GenAI in peer review and provides a basis for its responsible future implementation.
