A Hybrid Gabor–ViT Embedding with Gated Mamba Deep Learning Framework for Brain Tumor MRI Classification

Abstract

Brain tumor classification from MRI is an important task in medical image analysis. This study introduces a new hybrid deep learning method that integrates Gabor filtering for the improvement in texture feature extraction, the Vision Transformer (ViT) framework for extraction of the global context dependency and fine-grained spatial features, and a novel Gated Mamba to enhance feature classification efficiency. The framework was first evaluated in-distribution on the Kaggle dataset, which contains 5712 slices of brain MRI images from four different classes using accuracy, precision, recall, F1-score, AUC, Std, and 95% confidence intervals (CIs), with a mean image-level accuracy of 99.62%. Because the Kaggle dataset is known to be near-saturated and no patient identifiers are available, this in-distribution result is treated as a sanity check rather than a measure of clinical generalization; pHash deduplication was used to remove visually identical images, but this does not provide patient-level grouping, and residual patient-level leakage cannot be fully excluded. The central evaluation of this study is therefore a cross-dataset validation on the “figshare brain mri” dataset, which uses a different scanner and acquisition protocol and was never used during training and model selection; the proposed model on this dataset achieves 87.23% accuracy with reasonable confidence intervals, a decline of 12.39 percentage points relative to the in-distribution result that reflects the domain-shift gap typical of this task. The proposed hybrid approach performs competitively against established methods in-distribution, while the cross-dataset results highlight generalization challenges that motivate the failure analysis and limitations discussed in this paper.