Dual-branch cross-modal architecture with global-to-local feedback for radiographic image–text retrieval in chest X-rays

Abstract

Radiology reports are vital for accurate diagnosis and treatment planning, yet their manual generation is time-consuming and dependent on radiologist expertise, leading to delays and inconsistent clinical decisions. Medical image–text retrieval offers a scalable solution by enabling the retrieval of relevant prior cases and reports. However, existing models often rely on static embeddings, which limits their ability to capture fine-grained, clinically meaningful cross-modal relationships. This limitation is particularly critical in chest X-ray interpretation, where nuanced textual descriptions must align with subtle visual cues. We introduce a novel dual-branch retrieval framework that distinguishes between shared semantics and complementary features through a Synergy branch and a Difference branch, respectively. These branches are stabilized through orthogonal regularization, ensuring minimal redundancy while keeping complementary diagnostic cues. A Global-to-Local Feedback mechanism guides fine-grained local attention using global context, enhancing interpretability and clinical relevance. Evaluated on two benchmark datasets, our model achieves state-of-the-art retrieval performance across both image\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}text and text\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rightarrow$$\end{document}image tasks. Through hierarchical gated co-attention, our approach dynamically aligns image and text representations, addressing the limitations of static fusion and providing a foundation for interpretable, real-time retrieval systems that can accelerate and improve clinical decision support.