Skip to content
Book Open access

Reliable Evaluation of AI Assisted Peer Review

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 26 references

Abstract

Large language models (LLMs) are rapidly moving from experimental prototypes into operational peer review workflows, where they are used to polish, rewrite, synthesize, and generate reviews. This shift creates an urgent challenge for organizations that depend on expert assessment. Rather than asking only whether AI can improve peer review, this talk confronts a more fundamental question: can we reliably measure what makes a review useful, fair, and decision-relevant? The talk examines this question through the lens of AI-assisted peer review and shows that commonly used signals and LLM-as-a-judge evaluation often fail to capture substantive review quality. Drawing on real-world deployment experience at Reviewerly, we present case studies in which evaluation metrics reward surface-level improvements like polish and fluency while overlooking evidentiary content, correctness, and editorial usefulness. We then discuss practical lessons for building reliable evaluation systems in real peer review workflows. Although peer review is the central case study, the broader message applies to many industry and organizational settings where AI is being introduced into expert judgment workflows. Responsible automation requires more than stronger generation models. It requires evaluation methods that measure the qualities we actually want AI systems to improve.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.