VSF: A Multimodal Few-Shot Plant Disease Recognition Method in Agricultural Scenarios

Abstract

Identifying plant diseases presents challenges like small sample sizes, imbalanced classes, and intricate background noise, which constrain the generalizability and resilience of conventional deep learning methods that rely on single-modal images. To overcome these limitations, this study introduces a novel approach for recognizing plant diseases with limited examples, utilizing multiple modalities that combine visual and textual data to enhance the model’s discriminative power and generalization performance. First, the official definition of plant diseases is semantically enriched using large language models to automatically generate high-quality textual descriptions that include symptoms, affected plant parts, and agricultural context characteristics. Second, a visual–semantic multimodal fusion framework is developed, incorporating a Two-Stage Modality Fusion mechanism that projects semantic and visual features into a unified feature space to optimize cross-modal feature interaction. The experiments were performed on the PlantVillage dataset and the PlantDoc dataset to validate the N-way K-shot task. Results show that the proposed method achieved accuracies of 79.01% and 90.51% under the 5-way 1-shot and 5-way 5-shot tasks on the PlantVillage dataset, and 59.82% and 70.61% on the PlantDoc dataset. These results outperformed traditional unimodal and existing multimodal approaches. This paper provides an efficient and widely applicable multimodal approach to few-shot plant disease identification in agriculture.