Skip to content

HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews

Sep 2026 · 0 citations · 20 references
Computer Science

TL;DR

HalluPeer is introduced, a benchmark for detecting hallucinations in scientific peer reviews, providing aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization.

Abstract

The growing scale of academic peer review has motivated the use of Large Language Models (LLMs) as review assistants, yet LLMs can generate fluent but unsupported claims that undermine review reliability. Existing hallucination benchmarks are not designed for peer review, where verification requires grounding claims in long, technical papers. We introduce HalluPeer, a benchmark for detecting hallucinations in scientific peer reviews, providing aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Our pipeline induces a peer-review-specific hallucination taxonomy, identifies review contexts, and injects hallucinations with automated filtering. Experiments on 12K papers and 38K reviews show that existing detectors struggle to separate hallucinations from legitimate critique, while evaluation on authentic reviews demonstrates that HalluPeer-defined hallucination patterns occur in real peer reviews, highlighting the critical need for source-aware verification. Our project page can be found in https://github.com/Lin-TzuLing/HalluPeer.git

View source

Similar papers

Conference Open access Sep 2026

Mitigating Hallucinations in Natural Language Generation through Prompt Engineering: A Mechanism- Oriented Narrative Review

Large language models (LLMs) can generate fluent and confident responses that are factually incorrect, unsupported by evidence, or inconsistent with the source material. These hallucinations reduce the reliability of LLM-based question answering, summariza tion, dialogue, and information retrieval systems, especially w...

Jia-Nian Lin · 0 citations
#natural language process... Preprint Oct 2026

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and corresponding short reference answers. First, an LLM is used to generate answers to questions w...

J. Valjakka, Juhani Kivimäki, Juha Mylläri et al. · 0 citations
#machine learning Preprint Sep 2026

MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

This work introduces MISHAP-Bench, a comprehensive benchmark with 12,000 challenging open-ended question-audio pairs and a rigorous evaluation pipeline covering two hallucination categories, and proposes a groundedness judge that uses reference rubrics and judge prompts guided by human annotations.

Wen-Soi Zhi, Giulio Segalini, Jian-Jia Chen et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Leveraging Low-Level Symbolic Competences for Unsupervised Grounding in Hallucination Detection

This work investigates whether an LLM can use an alternative, low level, symbolic competence such as SQL for unsupervised hallucination detection in some high level task and finds that this approach improves on direct prediction and competes with state-of-the-art hallucination detection methods, while not requiring dom...

Renato Vukovic, Hsien-Chin Lin, Carel van Niekerk et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.