Hybrid transformer and CNN for hyperspectral image classification with spectral band selection

Abstract

Hyperspectral images (HSI) play a vital role in remote sensing applications such as surveillance and precision agriculture, yet they face challenges including limited labeled samples and class imbalance. This paper presents a new method for HSI classification by integrating spectral band selection with advanced deep learning techniques. First, we propose an optimized band selection method alongside data augmentation using crop-and-resize transformations to enhance feature diversity. Second, we introduce a hybrid convolution-transformer framework that combines Vision Transformer (ViT) and 2D-CNN architectures, supplemented with a sequence aggregation layer to mitigate overfitting on small datasets. This hybrid model effectively captures both spatial and spectral features while preserving critical spectral details. Finally, we employ a support vector machine for robust classification. Extensive experiments on three widely used HSI datasets demonstrate that our proposed ViT-CNN-BS framework achieves superior classification accuracy compared to state-of-the-art few-shot learning methods and traditional approaches. Beyond achieving higher accuracy, our approach also reduces computational complexity, making it suitable for resource-constrained applications. The findings highlight the potential of hybrid deep learning frameworks for advancing remote sensing analytics, with possible extensions to environmental monitoring, disaster management, and precision agriculture.