A segmentation-guided CNN–Vision transformer feature fusion framework for multi-class breast ultrasound image classification

Abstract

Breast ultrasound (BU) imaging is widely used for detecting breast abnormalities because it is cost-effective, non-invasive, and suitable for dense breast tissue. However, multi-class classification of BU images is considered a challenging task due to low contrast, speckle noise, and overlapping visual patterns between benign and malignant tumours. To address this issue, we develop a segmentation-guided CNN and Vision Transformer based feature fusion framework for efficient multi-class BU image classification. The framework first applies a lesion segmentation model to identify the region of interest by using Fusion-Enhanced Transformer (FET) Unet model. The FET Unet model uses CNNs and Swin Transformers integrated and constructs a UNet-like architecture. In the second step, CNN-based and Vision Transformer-based features are extracted from the segmented lesion regions. In the third step, the CNN and Vision Transformer features are fused to generate a robust feature representation. A deep neural network classifier comprising four dense layers with 1,024, 512, 256, and 128 neurons, respectively, followed by a softmax output layer, is then developed using the fused features to classify breast ultrasound images into benign, malignant, and normal categories. The proposed framework was evaluated using the publicly available Breast Ultrasound Images (BUSI) dataset, which contains 780 ultrasound images collected from women aged 25–75 years, including 437 benign, 210 malignant, and 133 normal cases. Numerical results show that the proposed framework showed 95.11% of accuracy, 95.66% of sensitivity and 97.63% of specificity and F1 score of 0.944644. Based on the classification accuracy, the effectiveness of the proposed hybrid framework is demonstrated in comparison with previously reported methods for multi-class breast ultrasound image classification.