UniMedSeg: Unified Multi-Modal Medical Image Segmentation via Uncertainty-Guided Encoding and Frequency-Spatial Dual-Prompt Decoding

Abstract

Text-driven medical image segmentation demands unified models based on multi-modal learning, capable of processing diverse imaging modalities while providing spatially reliable prediction estimates for clinical practice. Nevertheless, existing vision-language model solutions suffer from poor domain generalization and exhibit limited capability for uncertainty estimation to model the cumulative unreliability of cross-modal alignment across network layers. In this paper, we propose UniMedSeg, a unified vision-language framework for multi-modal medical image segmentation built upon frozen pre-trained vision-language backbones. Our framework incorporates modality-aware adaptation, cross-layer uncertainty-guided encoding, and a frequency-spatial dual-prompt decoder, wherein the progressive uncertainty propagation mechanism of the encoder suppresses misleading cross-modal features, whereas the decoder disentangles anatomical structure and boundary details in the frequency domain under dual textual prompts. Auxiliary losses further sharpen boundary representations and calibrate pixel-wise uncertainty maps. Extensive experiments demonstrate that UniMedSeg preserves competitive in-domain segmentation performance, improves domain generalization, and yields clinically useful spatial risk estimation. The source code is available at https://github.com/699zjl/UniMedSeg. Highlights We propose UniMedSeg, a unified vision-language model for five-modality medical segmentation that outputs segmentation masks and clinical-oriented pixel-level uncertainty maps. A cross-layer uncertainty propagation mechanism is designed to dynamically suppress unreliable vision-language alignment by recursively accumulating historical reliability signals. A frequency-spatial dual-prompt decoder disentangles anatomical structure and boundary details via FFT-based separation and branch-specific text-guided attention to reduce frequency-domain artifacts. Auxiliary boundary contrastive and uncertainty calibration objectives enhance boundary discrimination and enforce consistency between predicted uncertainty and segmentation errors. Extensive experiments on multi-modal medical datasets verify that UniMedSeg yields competitive in-domain performance and favorable out-of-distribution generalization capability.