MSCT-Trans: A Multi-scale Convolutional Neural Network Token Transformer for Interpretable Ultrasound Image Classification.
TL;DR
Multi-Scale CNN Token Transformer (MSCT-Trans), a lightweight and interpretable hybrid architecture for general-purpose ultrasound image classification, is proposed, which consistently outperformed CNN and Transformer baselines across accuracy, macro-F1 and area under the receiver operating characteristic curve, particularly under class imbalance and limited data regimens.
Abstract
Automated ultrasound image classification is increasingly important for clinical decision support in breast, thyroid and fetal screening. However, deploying deep learning models in such safety-critical settings demands not only high predictive accuracy but also transparency, interpretability and trustworthiness-properties that existing approaches address insufficiently. Convolutional neural networks (CNNs) capture local texture patterns but struggle with global contextual dependencies, while Transformer-based models offer long-range reasoning yet require large-scale training data and remain sensitive to ultrasound-specific noise, both limiting factors for clinical deployment. We propose Multi-Scale CNN Token Transformer (MSCT-Trans), a lightweight and interpretable hybrid architecture for general-purpose ultrasound image classification. MSCT-Trans extracts multi-scale feature maps from a pre-trained CNN backbone and converts them into a unified token sequence, enabling a Transformer encoder to model global dependencies and inter-scale interactions over semantically meaningful, noise-attenuated representations. To support clinical transparency, we conducted a two-part explainability analysis-Gradient-weighted Class Activation Mapping++ spatial localisation and softmax class probability breakdown-demonstrating that MSCT-Trans consistently attends to diagnostically relevant anatomical regions, produces well-calibrated confidence estimates and associates prediction errors with model uncertainty rather than over-confident mis-classification. Here we evaluated MSCT-Trans on three ultrasound benchmarks spanning breast (BUS-BRA + BUSI + UCLM), thyroid (TN5000) and fetal imaging. MSCT-Trans consistently outperformed CNN and Transformer baselines across accuracy, macro-F1 and area under the receiver operating characteristic curve, particularly under class imbalance and limited data regimens. The combination of strong predictive performance, spatially grounded interpretability and calibrated uncertainty estimation positions MSCT-Trans as a transparent and trustworthy foundation for ultrasound-based clinical decision support. Code: https://github.com/MohsinFurkh/MSCT-Trans.