MedFuseNet: Compact Dual-Stream Fusion of CNN and Vision Foundation Features for Medical Image Classification

Abstract

Medical image classification requires sensitivity to subtle local lesion morphology together with sufficient global contextual understanding of anatomical structure. Single convolutional neural networks (CNNs) emphasize texture and edge-level patterns but may underrepresent long-range relationships, whereas transformer-based vision foundation models (VFMs) provide broad semantic representations that may be less responsive to fine task-specific structures in small medical datasets. This study presents MedFuseNet, a compact dual-stream framework that combines a trainable CNN branch with a frozen VFM branch for two-dimensional medical image classification. The CNN stream learns dataset-adaptive local features, while the VFM stream supplies stable global semantic representations. The two feature vectors are fused after global pooling through direct concatenation and processed by a lightweight multilayer classification head. Experiments on the twelve 2D subsets of MedMNIST v2 show that the proposed fusion strategy provides robust performance across pathology, blood-cell, retinal, dermoscopic, radiographic, ultrasound, and organ CT tasks. The compact DINOv2-ViT-S/14 + ResNet-18 configuration achieved a mean AUC of 0.919 and a mean accuracy of 0.818. Ablation results further show that increasing VFM scale does not monotonically improve performance and that direct concatenation provides a more stable AUC–accuracy trade-off than adaptive weighting or attention-based fusion. These findings support a simple design principle for data-limited medical image classification: preserve complementary local and global representations and fuse them late without unnecessary coupling complexity.