Crop–Weed Multispectral Semantic Segmentation With Foundation Models and Hardness-Aware Learning

Abstract

Precision agriculture increasingly relies on accurate semantic segmentation of crops and weeds to enable targeted interventions such as selective weeding and yield optimization. However, conventional deep learning approaches often struggle to generalize across diverse field conditions, particularly when they are limited to standard visual spectrum imagery. In this work, we propose a multispectral semantic segmentation framework that integrates convolutional features, self-supervised transformer representations, and vision–language priors within a unified architecture. Spatial features extracted from a convolutional backbone adapted to multispectral input are combined with semantic representations from a self-supervised vision transformer, and fused through a multi-scale attention mechanism. In addition, a CLIP-guided cross-attention decoder exploits semantic priors to improve class-level discrimination. To further address class imbalance and challenging regions, we incorporate a hardness-aware learning strategy that adaptively re-weights the training objective based on pixel-wise hardness. Across five independent runs, the proposed framework achieves <inline-formula> <tex-math notation="LaTeX">$85.31 \pm 0.38\%$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$59.21 \pm 0.92\%$ </tex-math></inline-formula> mean Intersection over Union (mIoU) on the three-class and six-class WeedsGalore tasks, respectively, outperforming the best previously published methods by 2.41 and 3.69 percentage points. It also achieves F1-scores of <inline-formula> <tex-math notation="LaTeX">$0.875 \pm 0.002$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$0.709 \pm 0.005$ </tex-math></inline-formula> on the Rheinbach and Eschikon subsets of WeedMap, respectively, and <inline-formula> <tex-math notation="LaTeX">$86.63 \pm 1.47\%$ </tex-math></inline-formula> mIoU on CWFID. On the official PhenoBench test set, the proposed method reaches 87.65% mIoU, outperforming the baseline by 1.67 percentage points. Extensive experiments across multiple datasets demonstrate strong performance across both multispectral and RGB-based agricultural scenarios.