ITC-OVAD: An Image–Text Collaborative Strategy for Open-Vocabulary Aerial Detection

Abstract

Aerial object detection plays a crucial role in numerous applications. However, most existing methods focus on detecting predefined object categories, limiting their applicability in real-world open scenarios. In this article, we extend aerial object detection to open scenarios through image–text collaboration and propose ITC-OVAD, an end-to-end open-vocabulary detector for aerial imagery. Specifically, we first introduce an image-to-text alignment loss to replace the conventional category regression loss, thereby eliminating category constraints. Next, we propose a lightweight image–text collaboration strategy with bidirectional modulation between visual and textual modalities. Specifically, this strategy comprises three modules: text-guided feature enhancement (TG-FE) and visual-guided text refinement (VG-TR) in the encoder, and text-guided query enhancement (TG-QE) in the decoder, which together enrich visual features with class semantics and refine class embeddings with visual context, improving detection accuracy without incurring significant computational overhead. Extensive experiments demonstrate that ITC-OVAD consistently outperforms existing state-of-the-art (SOTA) methods across open-vocabulary, zero-shot, and traditional closed-set detection tasks. For instance, on the open-vocabulary aerial detection benchmarks DIOR, DOTA-v2.0, and LAE-80C, ITC-OVAD achieves 87.7 AP50, 53.8 mean average precision (mAP), and 23.7 mAP, respectively, surpassing the previous SOTA (LAE-DINO) by 2.2, 7.0, and 3.5 points. Meanwhile, ITC-OVAD achieves real-time inference at 34 frames per second (FPS) on an RTX 4090 GPU, approximately three times faster than LAE-DINO (10 FPS). The code will be released at https://github.com/GT-Wei/ITC-OVAD