COJEPA is presented, a self-supervised framework for volumetric brain MRI that combines a joint-embedding predictive architecture (JEPA) with a contrastive loss (CO), targeting two complementary properties: local predictivity and global discriminability.
Abstract
Self-supervised learning offers a compelling approach for medical imaging, where labeled data are scarce and acquisition costs are high. We present COJEPA, a self-supervised framework for volumetric brain MRI that combines a joint-embedding predictive architecture (JEPA) with a contrastive loss (CO), targeting two complementary properties: local predictivity and global discriminability. The model is trained without labels on T1-weighted structural MRI from two cohorts (HCP-YA and AABC, $N{=}2286$, ages 22 to 90), extending I-JEPA to 3D with foreground-aware block masking, a hierarchical convolutional patch embedding, and world-space sinusoidal positional encodings. We evaluate all three objectives across zero-shot twin retrieval, brain tumor segmentation (BraTS 2024), and age regression (OpenBHB). COJEPA achieves the best monozygotic twin recall at rank@1 (0.84), the best finetuning age MAE (2.55 years on OpenBHB 3.0T), and matches CO on BraTS whole-tumor Dice, demonstrating that the combined objective yields representations that are simultaneously discriminative and locally structured.
Joint-embedding predictive architectures (JEPAs) have primarily been developed for self-supervised representation learning. Denoising JEPA (D-JEPA) recently demonstrated strong generative capabilities on natural images, yet the applicability to 3D medical imaging remains unexplored. Building on the D-JEPA framework, we present Med-D-JEPA, a systematic adaptation and evaluation of joint-embedding predictive generation for 3D brain MRI. Med-D-JEPA operates on continuous latent tokens produced by a 3D KL-regularized adversarial variational autoencoder, and combines masked context prediction, representation-level alignment, per-token diffusion, and iterative next-set-of-token sampling. We evaluate unconditional and class-conditional generation quality on BraTS2019 and OASIS-1 datasets; downstream classification utility; and preliminary whole-tumor segmentation on BraTS2020. Across different generation settings, Med-D-JEPA achieves superior or competitive performance compared to several strong baselines on fidelity and diversity metrics. Compared to training with real samples, Med-D-JEPA-based synthetic pretraining improves classification AUC from 0.63 to 0.85 on BraTS2019 and from 0.78 to 0.87 on OASIS-1. In the segmentation study, pretraining on Med-D-JEPA samples improves Dice from 0.74 to 0.80 and reduces HD95 from 13.40 to 9.56 mm. These findings establish joint-embedding predictive generation as a promising direction for 3D medical image synthesis and encourage further research in this direction.
Meng Zhou, Wen-Hao You, Yu-Xin Chen et al.· 0 citations
Accurate brain age estimation from structural MRI serves as a sensitive biomarker of neurological health. We propose NeuroFusion, a unified framework that addresses two fundamental limitations of existing approaches: vanishing gradients in deep 3D architectures and poor generalization across clinical settings with limited labelled data. NeuroFusion combines (1) dual-objective self-supervised pretraining (SimCLR + MAE) on 42,000 unlabeled volumes, (2) a 3D Vision Transformer backbone, (3) a cross-modal fusion module integrating demographic metadata, and (4) deeply-supervised multitask learning with uncertainty-weighted loss balancing. On the OpenBHB benchmark (N=3,966 healthy controls), NeuroFusion achieves state-of-the-art brain age prediction (MAE = 2.84 years). Crucially, in few-shot adaptation to unseen sites, NeuroFusion maintains strong performance with only 5 labelled examples, demonstrating clinically relevant generalizatio
Molla Md Rony, Rahman Md Takibur, Md. Emran Hossain Dipu et al.· International Journal of Adv...· 0 citations
Self-supervised pretraining is central to 3D medical image analysis, where unlabeled CT volumes are abundant but expert annotations are scarce. Yet existing volumetric encoders often fail to preserve the coarse spatial and geometric structure that downstream reasoning depends on, limiting their performance on organ disentanglement, abnormality detection, and spatial understanding when paired with language models. We introduce Rad-JEPA 3D, a joint-embedding predictive framework that learns volumetric CT representations by predicting the latent features of a complete scan from a masked view. At its core is a hybrid H-Mamba encoder that fuses a Mamba state-space branch, which models inter-slice continuity through sequential scanning, with a grouped-query attention branch, which captures cross-plane spatial context, combined through a lightweight per-token router. To improve the quality of intermediate representations, we further propose Hidden States Orthogonal Regularization (HSOR), which aligns student-teacher hidden states and reduces feature redundancy throughout the encoder. This layer-wise regularization produces more consistent and discriminative volumetric representations, leading to improved performance on organ recognition and spatial reasoning tasks. Pretrained on approximately 120,000 CT scans, Rad-JEPA 3D attains state-of-the-art results despite its compact size: with only 4.0B total parameters, it achieves competitive results with state-of-the-art on closed-ended VQA and the best average spatial-reasoning score on the Spatial-Med benchmark. Ablation studies confirm that the hybrid block and HSOR contribute complementary gains, and that the induced spatial structure can substitute for raw language-model scale on volumetric reasoning tasks.
Deep learning has shown strong potential in medical image analysis, but most existing methods rely on large-scale annotations and a closed-world assumption that rarely holds in clinical practice. Although Generalized Category Discovery (GCD) has advanced rapidly on natural images, it remains underexplored in medical imaging. To address this issue, we propose MedXplore, a unified framework for reliable and unbiased medical GCD, optimizing from both perceptual and decision levels. Specifically, at the perceptual level, taking a frequency domain perspective, Frequency-SNR Adaptive Attention and Consistency (FAAC) performs learnable full-spectrum filtering and global-local energy contrast activation to not only highlight local abnormal signals relative to the global context, but also provide reliable semantic anchors for patch consistency learning. At the decision level, Adaptive Cosine-Angular Margin (ACAM) adjusts angular margins using semantic difficulty and feature confidence to balance intra-class compactness and inter-class separability. Together, the two modules improve lesion-sensitive representation learning and mitigate old-class bias. Experiments on multiple benchmarks show an average \textbf{8.5\%} gain in \textit{All} accuracy over the strongest competing methods. On Kvasir, MedXplore reduces false-old errors from 14.50\% to 0.80\%, demonstrating strong robustness under severe old-new ambiguity.
Jianwei He, Kailin Lyu, Junhao Dong et al.· 0 citations
This work proposes an image-domain dual-branch INR framework, termed I-FP-INR, which extends the original INR design by introducing an additional feature-processing branch, which aims to extract complementary feature embeddings to enhance the overall representation, thereby benefiting reconstruction.
Donghang Lyu, Marius Staring, Yiming Dong et al.· 0 citations
Introduction Accurate brain lesion segmentation in Magnetic Resonance Imaging (MRI) remains challenging due to heterogeneous lesion appearance, variable scales, and ambiguous boundaries. Methods We propose DBENet, a Dual-Branch Encoder Network for brain MRI lesion segmentation. The spatial and frequency branches first extract lesion-related features from complementary domains. The Spatial and Frequency Fusion (SFF) module then integrates these features into a hybrid lesion representation. In parallel, lesion-aware prompts are encoded by the Segment Anything Model (SAM) prompt encoder to generate prompt embeddings. Finally, the Multi-scale Attention Fusion (MAF) module takes the hybrid representation and prompt embeddings as inputs, and progressively integrates them across scales for contextual modeling and boundary refinement. Results Extensive experiments on the ISLES 2022 and BraTS 2018 datasets demonstrate that DBENet outperforms state-of-the-art approaches, achieving Dice scores of 0.8621 and 0.8261, respectively, while providing superior lesion localization and boundary delineation. Ablation studies confirm the complementary benefits of SFF and MAF. Discussion Furthermore, DBENet achieves a favorable balance between segmentation accuracy and computational cost, supporting practical implementation under moderate hardware constraints. Future work will extend DBENet to volumetric 3D segmentation. We will also investigate multimodal learning and foundation-model adaptation to improve robustness across diverse imaging protocols.
