Hybrid CNN–transformer architecture for content based medical image retrieval
TL;DR
A hybrid Neural Network-Transformer encoder trained on four complimentary datasets maintains competitive accuracy across all four datasets while producing more retrieval-relevant embeddings and much more consistent performance under degraded queries suggests that the model may serve as a promising foundation for future CBMIR research and potential clinical exploration.
Abstract
Clinicians can quickly find previous cases that are visually and semantically similar to a query image using content-based medical image retrieval (CBMIR); however, its practical implementation is hindered by multiple modalities, a lack of labeled data, and vulnerability to real-world artifacts like cropping and blurring. This research presents a hybrid Neural Network-Transformer encoder trained on four complimentary datasets: the COVID-19 Radiography Database, the Kvasir GI endoscopy dataset, and the ImageCLEFmed 2007 and 2009 radiology benchmarks. The model creates compact 128-dimensional embeddings that integrate particular local cues with more general anatomical contexts using a ResNet-50 convolutional backbone, a lightweight transformer head, and a gated fusion module. In order to improve robustness and cross-domain generalization, this study employs data augmentation that mimics real corruptions during training and evaluates retrieval under both clean and intentionally degraded queries. Using a unified methodology based on Precision@K, Recall@K, and mean average precision (mAP), the augmented hybrid model maintains competitive accuracy across all four datasets while producing more retrieval-relevant embeddings and much more consistent performance under degraded queries. This suggests that the model may serve as a promising foundation for future CBMIR research and potential clinical exploration.