This Review examines volumetric foundation models, language alignment and compression strategies, and agentic systems that extend MLLMs through planning, tools, memory, and workflow interaction, and introduces a Claim-Design-Validation framework to assess whether technical, workflow, and clinical claims are matched by appropriate design and validation.
Abstract
Advances in multimodal large language models (MLLMs) are extending radiological artificial intelligence (AI) beyond task-specific image analysis toward multimodal understanding and reasoning. Volumetric radiology, however, presents a fundamental representational mismatch: clinical interpretation often requires full-volume spatial context and acquisition-dependent quantitative information, whereas current MLLMs are commonly conditioned on selected two-dimensional (2D) images, compressed visual representations, or report-derived text. Reliable volumetric radiology AI therefore requires representations that preserve task-relevant three-dimensional (3D) information and systems that can access, verify, and integrate this information across clinical workflows. In this Review, we examine more than 200 publications through July 2026. We organize the literature around volumetric representation and multimodal understanding at the model level, agentic orchestration at the system level, and their links to clinical applications and evaluation. We review volumetric foundation models, language alignment and compression strategies, and agentic systems that extend MLLMs through planning, tools, memory, and workflow interaction. We distinguish settings in which selected 2D views or report-mediated reasoning may suffice from those that warrant native volumetric modeling. We also introduce a Claim-Design-Validation framework to assess whether technical, workflow, and clinical claims are matched by appropriate design and validation. Across the literature, native volumetric modeling and agentic capabilities depend on the spatial, quantitative, contextual, and workflow requirements of the intended task. Clinical credibility requires faithful volumetric representation, traceable system behavior, claim-aligned validation, and clearly defined human oversight in realistic workflows.
A definitive taxonomy of the medical VLM landscape is provided, tracing the evolution from early Contrastive Alignment and Generative MLLMs to the cutting-edge frontiers of Dense Pixel-Grounding, Sparse Mixture-of-Experts (MoE), and Reasoning-Incentivized (RL) architectures.
Taha Razzaq, Murtaza Taj, Asim Iqbal· Journal of Biomedical Inform...· 0 citations
Vision-language models (VLMs) represent an emerging class of multimodal artificial intelligence (AI) systems that integrate visual information with natural-language understanding and generation. In computational pathology, VLMs provide a framework for aligning histologic morphology from whole slide images (WSIs) with pathology reports, and other text-based knowledge sources. This review summarizes the technical foundations, major applications, evaluation strategies, and deployment considerations of pathology VLMs. Current pathology VLMs support a growing range of use cases, including image-text retrieval, label-efficient classification, visual question answering, abnormality localization, anomaly detection, report generation, and agentic workflow support. These capabilities are enabled by image encoders, text encoders or large language models, multimodal alignment strategies, and, in some systems, generative language components. Despite rapid progress, several barriers remain. Evaluation of pathology VLMs is constrained by limited domain-specific benchmarks, insufficient assessment of visual grounding, overreliance on text-based metrics, vulnerability to hallucination, and uncertain robustness under data shift. Clinical translation also requires validation across institutions, scanners, staining protocols, tissue types, and patient populations, together with workflow integration, regulatory oversight, data privacy, cybersecurity, and pathologist accountability. VLMs are therefore best viewed as assistive systems that may augment rather than replace pathologists. Responsible development will require close collaboration among pathologists, computational scientists, health systems, and regulatory stakeholders to ensure that VLMs improves pathology practice in a safe, interpretable, and clinically meaningful manner.
Rong Xia, Brian R Isett, Jie Chen et al.· American Journal of Patholog...· 0 citations
While Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in 2D medical image understanding, their extension to 3D volumetric imaging remains hindered by prohibitive annotation costs and dataset opacity. Current data formats, predominantly consisting of rigid Visual Question Answering (VQA) pairs or unstructured final clinical reports, typically fail to capture explicit clinical reasoning. To address this limitation, we introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm. Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process, translating global clinical priors into fine-grained, per-slice observations that are subsequently synthesized into an interpretable Chain-of-Thought (CoT). Crucially, this synthesized reasoning framework enforces essential clinical principles: sequential spatial tracking, multi-slice spatial awareness for artifact mitigation, and differential exclusion. To validate this approach, we instruction-tune a standard 2D-pretrained MLLM baseline using the synthesized data to enhance its volumetric comprehension. Comprehensive evaluations across multiple 3D medical benchmarks demonstrate that our method yields significant performance improvements over the 2D baseline. Furthermore, the resulting model exhibits robust spatial reasoning capabilities and rivals resource-intensive native 3D architectures, effectively bridging the performance gap. Ultimately, this data-centric strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training. The complete repository, including datasets and training workflows, is publicly available at https://github.com/2020420145009/hounsfield.
Zhuoyuan Fu, Zeshang Li, Yiqiong Zhang et al.· 0 citations
The exponential growth in medical imaging volumes necessitates scalable, reliable diagnostic support systems capable of augmenting clinical workflows. This article presents a systematic quantitative evaluation of state-of-the-art Multimodal Large Language Models (MLLMs) for radiology Visual Question Answering (VQA), a task requiring integrated visual perception and clinical reasoning. We benchmark five leading models — GPT5-Nano, Gemini 3 Flash, Qwen3-VL-8B, LLaVA Next, and Llama 3.2 Vision — on the VQA-RAD dataset under a rigorous zero-shot protocol with standardized prompts and comprehensive precision–recall–F1 evaluation. Our empirical analysis reveals that Gemini 3 Flash achieves superior balanced performance (F1 = 0.78, Accuracy = 0.78, Recall = 0.83), while Qwen3-VL-8B attains the highest precision (0.78) while also maintaining competitive recall. These outcomes demonstrate that general-purpose MLLMs can perform competitively with specialized medical models in tasks such as modality and organ recognition, but still struggle with abnormality detection and complex clinical reasoning. The findings reinforce that MLLMs currently serve best as assistive decisionsupport tools rather than autonomous diagnostic agents, and highlight the potential of retrieval-augmented and context-aware strategies for improving clinical reliability and interpretability.
Cristovão Pessoa Cândido, Matheus Alves de Oliveira Lima, C. de Souza Baptista et al.· International Journal of Sem...· 0 citations
Medical multimodal large language models (MLLMs) are increasingly expected to perform complex image understanding tasks, yet their reliability is often compromised by frequent errors in visual interpretation. To systematically trace these failures, we traverse the hierarchy from high-level clinical tasks down to fundamental visual perception. We therefore introduce Perception-Bench, a large-scale benchmark comprising 1.13 million samples that assesses medical MLLMs across six dimensions: attribute judgment, spatial grounding, spatial understanding, disease prediction, anomaly detection, and report generation, spanning both 2D and 3D radiology images. Our analysis on Perception-Bench reveals that existing MLLMs lack the ability to capture even the most basic lesion attributes, such as location, size, and density. This inability to ground clinical outputs in primary visual evidence reveals that the models'diagnostic unreliability is rooted in a critical but overlooked bottleneck in low-level visual perception. Motivated by this, we propose RadSight, a perception-driven MLLM built upon a dual 2D/3D encoder architecture that preserves native imaging spatial structures. RadSight formulates medical image understanding as a four-stage progressive process: visual-language alignment, fine-grained visual perception, clinical diagnosis, and diagnostic interpretation. The model is trained on an 8.37 million perception-oriented corpus using progressive curriculum learning. On Perception-Bench, RadSight consistently outperforms existing MLLMs across all six evaluation dimensions, with particularly strong gains in spatial grounding and clinical diagnosis. It also achieves consistent improvements on public 2D and 3D medical benchmarks, further demonstrating that robust low-level visual perception is a critical foundation for reliable clinical understanding. Code and model will be publicly available.
Jianqi Liu, Weiwei Cao, Wanxing Chang et al.· 0 citations
BACKGROUND
Large language models (LLMs) have emerged as powerful transformer-based systems capable of capturing long-range dependencies and complex semantic relationships in clinical language. In this review paper, we first examine the technical foundations of medical LLMs, including transformer architecture, attention mechanisms, training paradigms, and retrieval-augmented generation.
RESULTS
We then survey documented applications in spine surgery and spinal care, highlighting moderate guideline concordance (46-67%) for diagnostic support, automated generation of operative notes and discharge summaries for administrative workflows, LLM-assisted literature review and manuscript drafting for research support (with ~68% novelty accuracy), and translation of complex surgical concepts into patient-friendly materials at a seventh-grade reading level. We next explore emerging multimodal models that integrate text, imaging, laboratory, and genomic data via cross-modal attention, demonstrating superior performance in holistic diagnostic and prognostic tasks.
DISCUSSION
Finally, we discuss key implementation challenges, including model accuracy and hallucinations; computational, privacy, and regulatory constraints under HIPAA/GDPR; and bias mitigation, to outline strategies for safe, effective, and equitable deployment.
CONCLUSION
By mapping technical capabilities to clinical and research use cases, this review highlights the promise of LLMs in enhancing decision support, workflow efficiency, research productivity, and patient communication in spine care, while emphasizing the need for interdisciplinary collaboration, robust evaluation metrics, and governance frameworks that prioritize patient safety and equity.
Fabio Galbusera, Andrea Cina· European spine journal· 0 citations
