Adapting Medical and General-Purpose Vision-Language Models for ENT Endoscopic Report Generation

Abstract

Endoscopic report generation for Ear, Nose, and Throat (ENT) images remains challenging because Vision-Language Models (VLMs) suffer from domain shift and medical hallucination, while full fine-tuning is computationally expensive. We propose a unified framework that combines Retrieval-Augmented Generation (RAG) with Low-Rank Adaptation (LoRA) for ENT endoscopic report generation. Evaluated on the ENTRep dataset using three open-weight VLMs, RAG+LoRA achieves the best performance, improving BLEU-1 by 89% over zero-shot inference on MedGemma-4B-IT. We further show that retrieval alone outperforms LoRA on a general-purpose backbone, highlighting the value of external clinical knowledge. Our failure analysis localizes the remaining errors to retrieval quality rather than language generation, highlighting pathology-aware retrieval as a promising direction for future work.