APPLICATION OF LLM + ZERO-SHOT LARGE MODELS FOR FRUIT OBJECT DETECTION
TL;DR
A zero-shot annotation framework that integrates OWLv2, Google’s second-generation open-vocabulary vision model, with large language models to enable multilingual, natural language-driven fruit recognition in smart agriculture, providing scalable solutions for automated annotation, real-time monitoring, and large-scale data collection.
Abstract
Efficient and flexible agricultural image annotation is crucial for intelligent crop monitoring in smart agriculture, yet conventional detection models are limited by fixed class labels and require extensive manual annotations. This study presents a zero-shot annotation framework that integrates OWLv2, Google’s second-generation open-vocabulary vision model, with large language models (e.g., GPT-3.5, DeepSeek V1) to enable multilingual, natural language-driven fruit recognition in smart agriculture. A user-friendly interface was developed to support individual or batch image annotation with adjustable sensitivity to meet diverse field requirements. Experimental evaluations demonstrated the framework's strong generalizability and semantic understanding capabilities, allowing recognition of unseen fruit categories and attributes such as ripeness or color. The system significantly reduces annotation time and labor costs, while enhancing accessibility through natural language interaction. To ensure a robust evaluation of generalizability, a cross-domain protocol was employed using a novel dataset from 2025. Results showed that OWLv2 achieved an F1-score of 0.80 and an mAP of 0.8301, significantly outperforming the pre-trained YOLO11 (F1: 0.74, mAP: 0.60) in zero-shot scenarios. OWLv2 exhibited superior flexibility and required no task-specific dataset retraining, although its computational demands remain higher than lightweight models like YOLO11. Notably, while the LLM (DeepSeek) introduced a total one‑time API latency of 598.3 ms (called only once for processing multiple images). the actual core computational latency of OWLv2 was only 257.7 ms per image. Despite a total processing time of 891.9 ms (including visualization output), the framework demonstrates superior recall (0.9080) and semantic flexibility without retraining. These results verify the enormous application potential of OWLv2 and similar zero-shot models in agriculture, providing scalable solutions for automated annotation, real-time monitoring, and large-scale data collection.