PUNF: A Prompt-Driven Unified Unpaired Medical Image Translation With Normalizing Flow.
TL;DR
A prompt-driven unified unpaired medical image translation method with normalizing flow (PUNF), which learns the mapping from multiple domains to a common latent space through a single modality-adaptive conditional normalizing flow (NF), where it construct a modality-aware actnorm layer (MAAL) for achieving modality-dependent normalization.
Abstract
Synthesizing missing modality medical images is a critical and challenging task. To achieve multimodality translation in a single model, existing methods usually construct an implicit modality-shared space with auxiliary loss and modulate the translation process with one-hot modality code. In this article, we propose a prompt-driven unified unpaired medical image translation method with normalizing flow (PUNF). It learns the mapping from multiple domains to a common latent space through a single modality-adaptive conditional normalizing flow (NF), where we construct a modality-aware actnorm layer (MAAL) for achieving modality-dependent normalization. In addition, we leverage learnable prompts and modality-specific text embedding from contrastive language-image pretraining (CLIP) as external knowledge to generate multilevel dynamic prompts, providing powerful and robust guidance about modalities. Furthermore, we develop a gated prompt-guided feature encoder (GPFE) to modulate features with tissue-awareness. Extensive experiments on three public multimodality datasets demonstrate that our PUNF model can generate reliable translations and achieve state-of-the-art (SOTA) performance both quantitatively and qualitatively. Moreover, our PUNF also obtains consistently excellent results in generalization evaluation and downstream task. Ablation studies further verify the effectiveness of our designed modules.