Skip to content
Open access

A Multimodal AI Framework for Medical Education: Integrating Adaptive Image Retrieval, Fast Synthesis, and LLM-Based Clinical Auditing

Sep 2026 · Journal of Imaging · Vol 12 · 0 citations · 74 references
Medicine

TL;DR

An extended version of MIRAGE, a multimodal retrieval and generation system that utilizes a shared latent space to process medical queries by retrieving real images from the ROCO dataset, generating synthetic scans, and providing LLM-based clinical descriptions alongside dual-concept visual comparisons is presented.

Abstract

Access to reliable medical images is essential for clinical training. To address this need, this paper presents an extended version of MIRAGE, a multimodal retrieval and generation system that utilizes a shared latent space to process medical queries by retrieving real images from the ROCO dataset, generating synthetic scans, and providing LLM-based clinical descriptions alongside dual-concept visual comparisons. To overcome previous computational limits and the lack of clinical validation, we introduce three core enhancements: first, an Auto-α module to dynamically weight visual and textual similarities; second, the integration of LCM-LoRA to accelerate synthetic image generation; and third, an automated clinical auditor based on Gemini 2.5 Flash. Experimental results demonstrate that Auto-α improves retrieval accuracy for heterogeneous queries, reaching 38.83% Top-1 Recall over a 65,419-image gallery and outperforming nine fusion baselines evaluated under a unified configuration, with a controlled ablation attributing most of this gain to learning the weight rather than merely making it query-adaptive, while the LCM-LoRA module reduces computational costs by a factor of 12.5× in CPU environments, with a blinded radiologist evaluation confirming only a small drop in clinical quality. Furthermore, the clinical auditor achieves a 0.805 Pearson correlation against an expert radiologist, effectively correcting the systematic overestimation of traditional CLIP scores. Finally, the optimized platform is publicly deployed on Hugging Face.

Read PDF

Similar papers

Open access Aug 2026

Hybrid CNN–transformer architecture for content based medical image retrieval

A hybrid Neural Network-Transformer encoder trained on four complimentary datasets maintains competitive accuracy across all four datasets while producing more retrieval-relevant embeddings and much more consistent performance under degraded queries suggests that the model may serve as a promising foundation for future...

Namita Bhatt, Alongbar Wary, Ritika Kumari · 0 citations
Preprint Aug 2026

SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering

SeVeR is proposed, a selective visual exposure framework that compresses dense volumes into modality-wise prototypes and retrieves complementary multi-level evidence with change-aware gated attention during decoding, trained with a marginal-utility self-consistency objective that suppresses unhelpful retrieval.

Yao-Jun Hu, Danyang Tu, Yang Liu et al. · 0 citations

Towards Multimodal Retrieval-Augmented Generation for Medical Visual Question Answering

A novel multimodal RAG framework tailored for MedVQA is proposed, which leverages multimodal data, including medical images, reports, and generated captions, to provide more accurate clinical answers, and introduces a training paradigm that uses captions as auxiliary supervision, enhancing cross-modal alignment via con...

Mai A. Shaaban, M. Zarei, Adnan Khan et al. · 0 citations
Preprint Aug 2026

LocAnyMed: Vision-Language Grounding for Multimodal Medical Images

LocAnyMed-CoT-20K is derived, a rationale-augmented subset that connects anatomical context, visual observations, and spatial conclusions through structured reasoning and further improves cross-source generalization through fine-tuning.

Zi-Han Wang, Tong Liu, Zhi-Wei Wang et al. · 0 citations
Sep 2026

Prototype-guided Multimodal Retrieval for Knowledge-assisted Interventional Radiology.

Experimental results show that PRIMA consistently improves generation quality, question-answering accuracy and expert-rated clinical interpretability compared with existing RAG baselines, demonstrating the effectiveness of clinically grounded prototype-guided retrieval for multimodal knowledge assistance in interventio...

Jing-Xiong Li, Cheng-Lu Zhu, Yuxuan Sun et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Representation-guided in-context learning for medical image interpretation with multimodal large language models

RG-ICL is introduced, a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates, and indicates that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.

Min-Da Zhao, Fang-Yu Hu, Yan Luo et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.