ITR: Iterative Transductive Refinement with a Graph-Reliability Gate for Zero-Shot Agricultural CLIP Classification
TL;DR
ITR (Iterative Transductive Refinement), a parameter-free, training-free method that refines the frozen-CLIP zero-shot posterior over a whole unlabelled test batch by iterating, for two rounds, three established operators: k-nearest-neighbour (k-NN) label propagation on CLIP image features, lightweight LLM-free agriculture-aware prompts, and a Sinkhorn–Knopp doubly stochastic projection.
Abstract
Contrastive Language–Image Pre-training (CLIP) has become the dominant paradigm for zero-shot visual recognition, classifying images directly from textual class descriptions with no task-specific labelled data. This label-free ability is a natural fit for agricultural plant-disease and weed recognition, where expert annotation is costly and the space of crops, diseases, and weeds continually expands with new classes, so a classifier that labels an unlabelled batch of field images without re-training is attractive. Off-the-shelf zero-shot CLIP, however, falls well short on this fine-grained domain: agricultural imagery is under-represented in pre-training, classes differ only in subtle textural cues, and the posterior skews systematically toward a few generic labels. We present ITR (Iterative Transductive Refinement), a parameter-free, training-free method that refines the frozen-CLIP zero-shot posterior over a whole unlabelled test batch by iterating, for two rounds, three established operators: (i) k-nearest-neighbour (k-NN) label propagation on CLIP image features, (ii) lightweight LLM-free agriculture-aware prompts, and (iii) a Sinkhorn–Knopp doubly stochastic projection. On three agricultural fine-grained benchmarks (PlantDoc, VL-PAW, PP2021) with a single frozen OpenAI-CLIP ViT-B/16 backbone, ITR improves top-1 accuracy by +11.87, +10.64, and +9.31 pp (mean +10.60), outperforming recent training-free test-time methods—TDA, CuPL, WaffleCLIP, CALIP, the transductive EM-Dirichlet, and the multi-model OTFusion (CLIP + DINOv2). A central part of ITR is a graph-reliability gate that, measuring the image graph’s edge label-homogeneity per batch, applies the structural refinement only where the graph is reliable and otherwise leaves the prediction untouched; it contributes +1.66 pp mean (+4.87 pp on VL-PAW) at no cost to any dataset. At one iteration, the Sinkhorn step reduces to a transductive correction of CLIP’s class-marginal bias, identifying the source of the gain. These results show that a frozen-CLIP model can be turned into a competitive agricultural classifier at test time, with no training or labelled data, offering a practical route to open-vocabulary plant-disease and weed recognition under label scarcity.