Skip to content

Representation-guided in-context learning for medical image interpretation with multimodal large language models

Sep 2026 · arXiv.org · Vol abs/2609.24057 · 0 citations
Computer Science

TL;DR

RG-ICL is introduced, a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates, and indicates that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.

Abstract

Medical image interpretation is central to diagnosis and care, yet adapting general-purpose multimodal large language models (MLLMs) often requires resource-intensive domain-specific fine-tuning. Here we introduce representation-guided in-context learning (RG-ICL), a training-free inference framework that retrieves query-aligned demonstrations using frozen encoders, without task-specific parameter updates. Across eight datasets spanning histopathology, radiology and retinal fundoscopy, RG-ICL improved classification (mean gain 20 percentage points) and visual question answering (VQA) (mean gain 13 percentage points) over no-context and conventional ICL, approaching or exceeding training-based comparators. Which cases were retrieved mattered more than how many: 6 query-aligned cases outperformed up to 32 randomly selected ones, whereas fixed or random cases often reduced accuracy below baseline. For VQA, aligning reference cases with both image content and question intent produced further gains. These findings indicate that for medical image interpretation, curating which reference cases an MLLM sees is a practical alternative to retraining it.

View source

Similar papers

Preprint Sep 2026

A visual large language foundational model for medical image recognition using clinician-contributed online resources

Large language models (LLMs) have demonstrated strong capabilities across diverse domains, showing considerable potential in medicine. However, their application in medical settings remains limited by the scarcity of visual question answering (VQA) datasets that capture clinical reasoning and explicit image-text alignm...

Ling-Xuan Hou, Yu-Hua Xie, Yue Hu et al. · 0 citations
Open access Sep 2026

An Automated, Contamination-Controlled VQA Benchmark for Evaluating Vision-Language Models on 3D Oncology Imaging

Vision-language models (VLMs) are increasingly applied to medical imaging, yet public benchmarks may reward memorization over perception: their images and questions can enter pretraining corpora, and many items remain answerable from question text alone. We present an automated, agent-driven pipeline that builds multip...

Bo Liu, Han Gu, Xiang-Rui Li et al. · 0 citations
#natural language process... Preprint Aug 2026

Surgical Alignment in Knowledge Graph Training for Clinical Diagnosis with Large Language Models

A systematic study spanning five KG task formulations, three training paradigms, two KGs, and three base LLMs finds that at the task level, all paradigms improve over the non-finetuned baseline, but methods with comparable in-domain accuracy show substantially different knowledge transfer behavior.

Saksham Khatwani, He Cheng, M. Afshar et al. · 0 citations
Conference Sep 2026

Parameter-Efficient Fine-Tuning of InstructBLIP for Medical Visual Question Answering

Med-VQA systems help alleviate the diagnostic burden on radiologists by providing decision support to clinicians through the interpretation of clinical queries based on radiological images. Traditional models are classification-based and are limited to a restricted response space. This proves insufficient in complex me...

E. Balık, Mehmet Kaya · 0 citations
#artificial intelligence Preprint Sep 2026

A visual large language foundational model for medical image recognition using clinician-oriented social media

A FOundational LLM Trained on ThoughtMed-1M (FOLTMed), a scalable paradigm for advancing research on clinically grounded multimodal LLMs, achieved state-of-the-art performance across 42 medical VQA benchmark datasets, with a macro accuracy of 85.4%, and generated more clinically coherent responses on the ThoughtMed-1M...

Ling-Xuan Hou, Yu-Hua Xie, Yue Hu et al. · 0 citations
Open access Sep 2026

A framework for extraction of clinical information from radiological mammography reports using large language models and retrieval augmented generation

A structured Retrieval-Augmented Generation (RAG) framework that leverages annotated mammography reports as in-context demonstrations, enabling joint named entity and relation extraction without task-specific fine-tuning is proposed and achieves performance comparable to fine-tuned BETO models in low-resource settings,...

Eduardo Godoy, J. De Ferrari, Sofia Lazo et al. · 1 citation · ⚡1

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.