Variational Autoencoders for Medical Image Segmentation: A Review

Abstract

Medical image segmentation plays an important role in healthcare because it transforms raw scans into useful information for diagnosis and treatment planning. Despite the strong performance of deep learning, most supervised segmentation methods require large amounts of pixel-level annotations and may become less reliable when imaging modalities are incomplete or when prediction uncertainty is not properly considered. Variational autoencoders (VAEs) are deep learning models that were proposed to address some of these limitations, as they can learn probabilistic latent representations capable of capturing meaningful structure in medical images. This review provides a structured survey of how VAEs are applied to medical image segmentation and organizes them around five roles: generating synthetic samples for data augmentation, integrating variational representations into segmentation networks for representation learning and regularization, localizing pathology through reconstruction-based anomaly detection, fusing multiple modalities in a shared latent space for missing-modality segmentation, and learning anatomical shape priors for segmentation correction and quality control. In this work, the reviewed literature includes both approaches where the VAE directly contributes to the segmentation and related applications that support the segmentation pipeline, such as data generation, preprocessing or weak localization. We also summarize the main datasets and evaluation metrics used in the field and discuss current challenges, including blurry reconstructions, the use of hybrid architectures in several reviewed studies, the strong concentration of studies on brain MRI, the limited use of uncertainty, and the possible complementary roles of VAEs alongside diffusion and foundation models. Overall, the studies reviewed here suggest that VAEs can serve as parts of larger systems, where they provide structured representations and help handle missing modalities. Conditional models trained to predict masks can also generate multiple plausible segmentations, which may reflect variation among annotations when the training data capture it.