Multimodal Cross-Scene Hyperspectral Image Classification Guided by Dual-Agent Priori Knowledge

Abstract

Semantic information is increasingly used to improve the performance of cross-scene hyperspectral image (HSI) classification. However, existing methods have difficulty fully exploiting the prior knowledge of large language models (LLMs) in textual information and usually rely on static template prompts. Moreover, they remain insufficient in modeling the complementary information between images and text. To address these issues, this article proposes a two-agent prior-guided multimodal network (TPMNet) for cross-scene HSI classification. First, a cross-domain semantic modeling mechanism guided by dual-agent priors is introduced. In this mechanism, source-domain and target-domain LLMs generate their respective prior texts. These priors are updated using similarity relationships with source-domain and target-domain image features, together with a memory-guided mechanism. As a result, dynamic textual representations with cross-domain perception capability are obtained. Second, a bidirectional multimodal cross-domain feature alignment and enhancement (BMAE) strategy is designed. Bidirectional cross-modal attention interactions are introduced between images and text to capture complementary discriminative representations across modalities. Meanwhile, a gating mechanism is adopted to select features that are highly relevant to the task, thereby improving the effectiveness of multimodal cross-domain features. Finally, a multiview consistency constraint learning (MCCL) method is proposed. Multilevel augmented views are constructed for target-domain samples. Multiple classifiers are designed to enforce joint consistency across different views, which enhances the classification capability of the model on target-domain data. Experimental results show that the proposed method achieves better performance than seven advanced algorithms on multiple cross-scene hyperspectral classification tasks.