FK-Net: Frequency-Aware and Kernelizable Mamba-Transformer for Multispectral and Hyperspectral Image Fusion.

Abstract

Multispectral and hyperspectral image fusion (MHIF) is dedicated to fusing high-resolution multispectral images and low-resolution hyperspectral images (LR-HSIs) for conversion into high-resolution hyperspectral images. Nevertheless, the prevalent methods exhibit limitations in capturing the interactions between spatial and spectral features. This inadequacy results in the suboptimal utilization of spatial-spectral information from high-resolution multispectral image (HR-MSI) and LR-HSI, as well as the ineffective preservation of their distinctive characteristics. To address this issue, we propose a frequency-aware and kernelizable mamba-transformer network (FK-Net) for MHIF. Specifically, we first design a frequency-aware dual-mamba (FADM) fusion module, which uses the frequency information of two images dynamically extracted by Fourier transform to supervise and guide the scanning process and effectively realizes the cross-modal information fusion of HR-MSI and LR-HSI. Furthermore, we introduce a robust and spectral-friendly similarity metric, i.e., the spectral correlation coefficient (SCC) of the spectrum, to replace the original attention matrix and incorporate inductive biases into the model to facilitate training. Built upon it, we further utilize the kernelizable attention technique with theoretical support to form a novel efficient SCC-kernel-based self-attention (ESSA) and reduce attention computation to linear complexity. The proposed FK-Net effectively integrates the complementary spatial-spectral information from both modalities while preserving their inherent uniqueness, enabling the generation of more natural high-resolution images without significantly increasing computational complexity. The qualitative and quantitative experiments demonstrate the superiority of the proposed method over the state-of-the-art (SOTA) approaches.