Aug 2026· PeerJ Computer Science· 0 citations· 43 references
TL;DR
The proposed framework offers a practical and scalable approach to mitigating hallucinations without requiring task-specific fine-tuning, highlighting the potential of retrieval-augmented approaches for trustworthy artificial intelligence (AI)-assisted healthcare applications.
Abstract
The integration of large language models (LLMs) into healthcare is hindered by their tendency to generate hallucinations, a critical issue in medical question answering (MedQA). Retrieval-Augmented Generation (RAG) frameworks address this limitation by grounding LLM outputs in external, verifiable knowledge sources. In this study, RAG pipeline implemented without fine-tuning the base model is evaluated to enhance factual fidelity in MedQA. GPT-4 is integrated with a dense passage retrieval module based on Facebook AI Similarity Search (FAISS) and the BAAI/bge-small-en embedding model, and compared against a non-retrieval GPT-4 baseline.Evaluation is conducted on a held-out set of 1,000 questions from the MedQuAD dataset using widely adopted computational evaluation metrics, including BLEU-1 to BLEU-4, ROUGE-L, METEOR, text-level F1, Exact Match (EM), and embedding-based semantic similarity metrics such as BERTScore and SBERT. The RAG-enhanced configuration consistently outperforms the baseline across all evaluation metrics, with notable improvements including an increase in BLEU-4 from 0.3015 to 0.4224 and BERTScore from 0.8487 to 0.8691, indicating improved contextual coherence and semantic alignment. Qualitative analysis further indicates fewer omissions and more accurate use of domain-specific terminology. To assess generalization, cross-dataset experiments were conducted using the PubMedQA benchmark. The results reveal that while RAG significantly improves performance in in-domain settings, its effectiveness decreases under domain shift when the retrieval
corpus
is not well aligned with the target question distribution. Additional experiments with varying retrieval depths (k = 1, 3, and 5) demonstrate that increasing the number of retrieved passages can improve semantic similarity by providing richer contextual evidence. These findings provide empirical evidence that retrieval augmentation enhances both the factual reliability and semantic quality of LLM-generated answers in MedQA. The proposed framework offers a practical and scalable approach to mitigating hallucinations without requiring task-specific fine-tuning, highlighting the potential of retrieval-augmented approaches for trustworthy artificial intelligence (AI)-assisted healthcare applications.
This work extends PubHealthBench, a question answering benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented setting and systematically evaluates retrieval and generation choices, and introduces a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency.
Felix Feldman, Joshua Harris, Timothy Laurence et al.· 0 citations
Background/Objectives: Large language models (LLMs) show promise for clinical decision support, yet their accuracy in interpreting specialized medical guidelines remains uncertain. Retrieval-augmented generation (RAG) may enhance performance by grounding responses in authoritative knowledge bases. This study aimed to compare the accuracy, comprehensiveness, and safety of RAG-enhanced versus standard LLMs for answering clinical questions derived from the German S3 guideline for oral cavity carcinoma. Methods: We conducted a prospective, single-blind benchmark study evaluating six LLMs: one RAG-enhanced model (Custom GPT with guideline access), one consensus-based model (ConsensusGPT), and four standard models (DeepSeek-V3.2, Mistral Small 3.2, Qwen3-Next-80B, GPT-OSS-120B). Fifty clinical questions covering 17 guideline domains were presented to each model three times, yielding 900 evaluations. Three expert reviewers assessed responses using 5-point Likert scales for accuracy, comprehensiveness, and clarity, under a single-blind procedure, the effectiveness of which was tested by a pre-specified manipulation check. We then ran a paired within-model experiment in which each base model was queried with and without guideline access through a transparent, openly released retrieval pipeline, and scored every response with a condition-blind automated judge alongside deterministic retrieval metrics computed from the logs. Secondary outcomes included hallucination rates and guideline citation behavior. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Results: In a paired within-model design that held each base model fixed, adding transparent guideline retrieval improved accuracy—significantly in the three weaker open-weight models (Mistral, Qwen3, and GPT-OSS) and directionally in the already-strong DeepSeek and GPT-5 bases. Because a pre-specified blinding check found that experts could still identify retrieval-augmented answers with 98.5% accuracy, we anchored causal interpretation on measures that do not depend on the human raters, ranked by their independence: deterministic, log-derived retrieval metrics first, and then an automated, condition-blind LLM judge, whose agreement with the experts (Spearman ρ = 0.81, 95.7% within-one agreement) establishes shared calibration rather than independence from their bias. Deterministically from the retrieval logs, citation groundedness rose from 0% to 51–89% and retrieval recall@5 was 92%. On the judge, content-level hallucination fell from 42% to 4% and accuracy rose by a pooled +0.64 points (95% CI 0.47–0.80); the accuracy gain persisted after adjustment for response length (+0.48, 95% CI 0.22–0.73), which retrieval shortened rather than lengthened. The accuracy gain was large for weaker base models and small or non-significant for already-strong ones, whereas the hallucination and auditability gains were consistent across all models. The human ratings reproduced the judge’s accuracy effect (+0.61, 95% CI 0.49–0.74), and GPT-5 run through the transparent pipeline showed no significant difference from the proprietary Custom GPT (judge accuracy 4.48 vs. 4.58). Conclusions: Guideline retrieval yields a reproducible, largely base-independent improvement in the safety and auditability of LLM answers to clinical guideline questions, with accuracy gains concentrated in weaker base models. Because retrieval-augmented answers are recognizable to experts, rigorous evaluation should rely on rater-independent measures, and residual hallucination continues to require human oversight.
Andreas Vollmer, Lara Schorn, Felix Schrader et al.· Diagnostics· 0 citations
—This paper proposes a novel approach which combines small language models with retrieval-augmented generation in medical question answering to provide accurate and comprehensible information. Our method extracts relevant evidence from external knowledge and converts them into vector embeddings which are used for high-fidelity semantic retrieval. The small language model subsequently synthesizes the retrieved evidence into fluent, context-aware responses. Extensive experiments conducted on the PubMed benchmark dataset, the results show competitive performance to larger language models while being far more suitable for deployment in resource-constrained environments. Moreover, the proposed method supports efficient domain knowledge updates without the need for extensive re-training. Our implementation is available in the following GitHub repository: https://github.com/LeoBaoNguyen12/RagSLM-MQA.
B. Nguyen, Quang-Hung Le· Journal of Communications So...· 0 citations
This study implements a Retrieval-Augmentation Generation approach for automatic fact-checking of health claims using the PubHealth dataset, and compares its performance against a pure generative model.
A novel multimodal RAG framework tailored for MedVQA is proposed, which leverages multimodal data, including medical images, reports, and generated captions, to provide more accurate clinical answers, and introduces a training paradigm that uses captions as auxiliary supervision, enhancing cross-modal alignment via contrastive learning.
Mai A. Shaaban, M. Zarei, Adnan Khan et al.· 0 citations
In medical multiple-choice question answering (MCQA), Retrieval-Augmented Generation (RAG) can supplement the domain knowledge of language models (LMs). However, since vanilla RAG indiscriminately utilizes retrieved documents, it can degrade LM performance. To address this, we propose MedJudgeRAG. Our framework represents retrieved documents as a dynamic knowledge graph (KG) composed of entities and relations. For each option, the model judges an evidence verdict from the retrieved documents and the KG. Based on the verdict combination, the model determines a knowledge utilization strategy to reason toward the final answer. These capabilities are trained via supervised fine-tuning using structured reasoning traces generated by a teacher LM. The training employs a weighted cross-entropy loss that differentially weights the KG and reasoning segments. Experiments on two medical MCQA benchmarks demonstrate that MedJudgeRAG consistently outperforms both vanilla RAG and parametric baselines. Furthermore, ablation analysis reveals that the dynamic KG is more effective as graph-conditioned supervision at training time than as an explicit output at inference time. Our code is available at https://github.com/hyu-amllab/medjudgerag, and the generated reasoning traces are released at https://huggingface.co/datasets/youarethewon/medjudgerag.
Seongwon Seo, Seung-Hwan Cho, Young-Min Kim· 0 citations