Jul 2026· Journal of Artificial Intelligence and Data Science· Vol 2, pp. 16-23· 0 citations
TL;DR
This study implements a Retrieval-Augmentation Generation approach for automatic fact-checking of health claims using the PubHealth dataset, and compares its performance against a pure generative model.
Abstract
The rapid growth of digital platforms has accelerated public access to online health information, but it has also become a primary channel for the spread of health misinformation. Large Language Models (LLMs) have been widely adopted to address this problem through pure generative approaches; however, they frequently produce convincing yet inaccurate information, a phenomenon known as hallucination. This study implements a Retrieval-Augmentation Generation approach for automatic fact-checking of health claims using the PubHealth dataset, and compares its performance against a pure generative (Non-RAG) model. The proposed pipeline combines a FAISS-based semantic retriever, CrossEncoder re-ranking, and a FLAN-T5 generator to classify each claim as Fact or Myth while supplying relevant supporting evidence, and is integrated into a Gradio-based interface for claim verification and dataset expansion. The models were evaluated using accuracy, precision, recall, F1-Score, Cohen’s Kappa, faithfulness, context precision, and answer relevancy across four scenarios of question and claim normalization. The experimental results show that the RAG model outperformed the Non-RAG model on the classification metrics, with the best configuration achieving an F1-score of 0.7823, an accuracy of 0.7857, and a Cohen's Kappa of 0.5532, whereas the Non-RAG model suffered from extreme bias toward the Fact label and failed to detect the Myth label, reaching an accuracy of only 0.4211. Claim normalization was found to substantially improve classification accuracy, while the low faithfulness scores of the RAG model reflect a limitation of the metric on short binary answers rather than hallucination. These findings confirm that the retrieval and augmentation stages contribute significantly to producing more accurate, transparent, and evidence-based verification of health claims.
The proposed framework offers a practical and scalable approach to mitigating hallucinations without requiring task-specific fine-tuning, highlighting the potential of retrieval-augmented approaches for trustworthy artificial intelligence (AI)-assisted healthcare applications.
Biomedical fact-checking systems must do more than predict whether a claim is supported, contradicted, or unaddressed: they should also produce evidence that is faithful, complete, and useful for verification. We study this evidence-generation setting on CARE-XAI, a unified benchmark spanning five biomedical and health fact-checking sources. We compare base instruction LLMs, PubMed retrieval-augmented LLMs, fine-tuned LLMs, label-only LLMs, and biomedical encoder classifiers under a shared evaluation protocol. Biomedical classifiers remain strongest for verdict-only prediction, while fine-tuned LLMs are the strongest evidence-generating systems. PubMed retrieval is mixed: it helps PubMed-aligned sources such as PubMedQA and SciFact, but can distract models on broader public-health claims. We introduce Bio-GRACE, a gold-reference-normalized diagnostic for measuring whether retrieved evidence recovers the decision benefit of reference evidence. Bio-GRACE shows that retrieval utility is source-dependent, motivates selective retrieval, and exposes why retrieval recall and lexical evidence overlap are insufficient for biomedical fact-checking.
The rapid spread of false and misleading health information through digital platforms has become a major public health challenge, particularly during infectious disease outbreaks where delayed verification can influence public behaviour and hinder effective disease control. Although recent advances in automated health misinformation detection have shown encouraging results, most existing approaches rely heavily on global biomedical resources and often fail to capture the local context needed to verify claims in developing countries. This study presents a retrieval-augmented transformer framework designed to verify health-related claims using trusted evidence from the World Health Organization and the Nigeria Centre for Disease Control and Prevention. The framework combines semantic evidence retrieval with transformer-based classification to determine whether a claim is true, false, or misleading. To evaluate the proposed approach, a manually annotated dataset of 67 verified health claims covering coronavirus disease, Lassa fever, cholera, measles, and monkeypox was compiled from Nigerian fact-checking sources. Three transformer models and a retrieval-augmented configuration were evaluated. The Bidirectional Encoder Representations from Transformers model achieved the best performance, with an accuracy of 71% and a weighted F1-score of 0.66. Although retrieval augmentation did not improve classification performance because the current evidence repository was limited in size and coverage, the findings highlight the importance of comprehensive and authoritative knowledge sources for reliable health misinformation verification. The proposed framework provides a practical foundation for developing context-aware and evidence-driven health misinformation verification systems for Nigeria and other resource-constrained settings.
Isah M. Bukar, Bala Mairiga Abduljalil, Bashir Saleh Maina et al.· 0 citations
Automated fact-checking is essential for ensuring the reliability of public health information, yet the biomedical domain poses unique challenges. Validating biomedical claims requires rigorous interpretation of scientific literature, assessment of retrieved evidence, and comprehensive justification toward the conclusion. Although Large Language Models (LLMs) enhanced by Retrieval-Augmented Generation (RAG) and agentic search perform automated fact-checking in a retrieve-then-verify paradigm, current methods still output isolated prediction labels, lacking explanatory depth and offers limited utility for human understanding. To bridge this gap, we introduce an LLM-based agent named BioCheck Agent that generates structured biomedical fact-checking reports with agentic search. Rather than merely outputting supported or refuted labels, our agent synthesizes final conclusions with retrieved evidence and rigorous analysis. To ensure domain-specific accuracy, BioCheck Agent exclusively searches high-quality scientific literature in PubMed, utilizing advanced Boolean search operators. Recognizing that direct prompting often results in hallucinations and low-quality reports, especially for lightweight open-source models, we further propose the Evidence-Grounded Group Relative Policy Optimization (EG-GRPO) to perform reinforcement learning on BioCheck Agent with a task-specific reward that incentivizes advanced search behavior and high-quality evidence retrieval while penalizing hallucinations. Our experimental results show that compared to the base model Qwen3.5-4B, BioCheck Agent with EG-GRPO improves label prediction accuracy on SciFact by 9.95%. Furthermore, it achieves a 3.7% higher evidence quality score and a 19.63% lower evidence hallucination rate, demonstrating its ability to generate biomedical fact-checking reports with improved accuracy and quality.
Findings support RAG as a promising approach to improve the accuracy, consistency and safety of LLM-driven DMHIs through a controlled comparison of RAG-enabled versus RAG-disabled modes.
A. Gupta, Akshat Surolia, Shubham Mishra et al.· 0 citations
This work extends PubHealthBench, a question answering benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented setting and systematically evaluates retrieval and generation choices, and introduces a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency.
Felix Feldman, Joshua Harris, Timothy Laurence et al.· 0 citations