GroundMed-SAM: Prompt-based Zero-shot Medical Image Segmentation

Abstract

Medical image segmentation is a key component of computer-aided diagnosis and treatment planning. Despite substantial progress in deep learning–based models, most existing approaches depend heavily on large annotated datasets and often fail to generalize across heterogeneous clinical environments, limiting their deployment in real-world settings characterized by domain shifts and scarce expert annotations. This paper presents a zero-shot learning framework named GroundMed-SAM for medical image segmentation. The framework integrates GroundingDINO for prompt-based region localization and MedSAM for mask generation. To address the weak alignment between visual features and medical semantics in GroundingDINO, which is pretrained on general domain image-text pairs, we introduce learnable medical text embeddings that explicitly parameterize domain-specific terminology in a continuous semantic space. These embeddings are optimized during training to better align medical concepts with visual representations, thereby strengthening text-image correspondence and improving detection-guided segmentation. The proposed framework preserves true zero-shot capability, enabling segmentation of previously unseen anatomical structures without task-specific labels. Extensive experiments on multiple public datasets across diverse modalities and clinical contexts demonstrate that our method achieves competitive segmentation performance in-domain while exhibiting superior robustness under cross-domain evaluation. Although supervised baselines outperform the proposed framework by only 3–5% on in-domain datasets, they experience substantial performance degradation when evaluated on unseen domains. Additionally, the framework achieves an AUC of 98.9 in endoscopic polyp detection, highlighting the effectiveness of the proposed medical-aware textual embeddings in guiding region localization. These results demonstrate the effectiveness of the proposed framework in improving cross-domain generalization for medical image segmentation with limited annotations.