PE-MedSAM2: Parameter-Efficient Adaptation of MedSAM2 for 2D Medical Image Segmentation.
TL;DR
Multi-dimensional lightweight modules can reconcile segmentation quality with strict computational budgets when adapting video-centric foundation models to 2D clinical data.
Abstract
Objective
Foundation models such as MedSAM2 achieve strong zero-shot segmentation, but standard fine-tuning updates 11.73 M parameters, limiting deployment in resource-constrained clinical environments. Existing parameter-efficient fine-tuning (PEFT) methods typically focus on reducing storage cost, with less attention to computational complexity and inference latency. We aim to adapt MedSAM2 to 2D clinical modalities with near-transparent overhead across parameters, FLOPs, and latency.
Methods
We propose PE-MedSAM2, a lightweight adaptation framework organizing four complementary modules along two orthogonal axes (channel vs. spatial; feature enhancement vs. computation reduction): a Low-Rank Adapter (LRA) for domain transfer with 0.002 M parameters; a Parameter-Free Feature Enhancement (PFFE) module that uses fixed multi-scale gradient operators to extract high-frequency spatial priors without learnable parameters; an Ultra-Lightweight Adapter (ULA) that decouples spatial and channel transformations via depthwise separable convolutions; and a Dynamic Sparse Attention (DSA) module that concentrates attention on gradient-guided salient tokens.
Results
Across five RGB- like 2D benchmarks spanning polyps, skin lesions, and cell nuclei, PE-MedSAM2 attains the highest Dice similarity coefficient on four and the best average surface distance on four of five; on a sixth benchmark, chest radiography, it again improves over its MedSAM2 baseline. Relative to MedSAM2, the framework adds only 0.294 M trainable parameters, 0.89 G MACs (+0.7%), and roughly 1.2 ms latency.
Conclusion
Multi-dimensional lightweight modules can reconcile segmentation quality with strict computational budgets when adapting video-centric foundation models to 2D clinical data.
Significance
PE-MedSAM2 enables deployment of foundation-model-based segmentation in resource-constrained clinical settings while maintaining competitive contour fidelity. Code is available at https://github.com/Yexika/PE-MedSAM2.