LightFormer-3D: a lightweight hierarchical CNN-transformer hybrid for 3D medical image segmentation

Abstract

Brain tumor segmentation from 3D MRI is critical for treatment planning, yet deploying accurate Transformer-based models in clinical settings remains challenging due to prohibitive computational costs. State-of-the-art methods often exceed 90M parameters and require seconds per inference, precluding real-time use on resource-constrained hardware. We propose LightFormer-3D, a lightweight hierarchical CNN-Transformer hybrid that applies a co-design principle to jointly optimize patch embedding, self-attention, and multi-scale feature fusion, ensuring that efficiency gains compound across all stages. Specifically, a Depthwise Separable Convolution Patch Embedding (DSC-PE) reduces embedding parameters by 70%; a Multi-head Self-Attention with Spatial Sequence Reduction (MSA-SR) compresses self-attention complexity from O(N 2 ) to O(N 2 /r 3 ); and a Lightweight Scalable Feature Fusion (LSFF) module adaptively weights multi-scale features using only five parameters. Evaluated on the BraTS 2017 and BraTS 2021 benchmarks, LightFormer-3D achieves a mean Dice score of 87.0% on BraTS 2021 with 1.78M parameters and 9.3 GFLOPs, surpassing UNETR (80.2% Dice, 92.5M parameters) by 6.8 pp and the best competing lightweight method, U-Net_ASPP_EVO (86.0%), by 1.0 pp, while delivering 10× faster inference than UNETR (0.51s vs. 5.41s per case on an NVIDIA RTX 4060). Five-fold cross-validation (86.7% ± 0.2%) confirms stable generalization. Ablation studies validate each module's contribution, and zero-shot external validation on the independent MU-Glioma-Post dataset (203 post-treatment cases) confirms robust whole-tumor localization (WT Dice 85.0%) under severe domain shift. LightFormer-3D establishes that co-designed lightweight modules can simultaneously improve accuracy and efficiency, achieving a new Pareto-optimal point for brain tumor segmentation.