Open-vocabulary object detection based on fine-grained alignment

SRA-Det: Learning Omni-Grained Open-Vocabulary Detection Beyond Category Names

YOLO-TVP: Real-time open-vocabulary object detection with Semantic-Target Soft Cross-Entropy and text-visual prompts

Open-vocabulary object detection (OVD) models generally leverage vision-language pre-trained models to recognize novel categories via arbitrary text prompts. Nevertheless, their performance is restricted by two core limitations: semantic discontinuity arising from hard binary supervision in contrastive learning, and the inflexibility of single prompts to convey complex detection intents in practical scenarios. To tackle these issues, this paper proposes YOLO-TVP (Text–Visual Prompt), an efficient OVD framework built on the YOLO architecture, with two key designs. First, a Semantic-Target Soft Cross-Entropy (ST-SoftCE) loss is introduced. It constructs semantic target distributions from inter-class similarities in the shared text embedding space for open-vocabulary inference and supervises the detector's classification branch. This design embeds semantic relevance into supervision and enhances fine-grained discrimination among semantically similar categories. Second, a unified class-prompt embedding interface is developed to support both textual and visual prompts. Text prompts are projected into the shared prompt space, while visual prompts are formed by learnable weighted fusion of CLIP semantic priors and backbone visual features, eliminating the need for multi-modal prompt concatenation during inference. Experiments validate the effectiveness: on Flickr30k image-text retrieval with ResNet-50, ST-SoftCE improves Recall@1 by 9.56% over standard cross-entropy. For open-vocabulary detection trained on COCO+Flickr30k and evaluated on LVIS, ST-SoftCE delivers a 13.15% relative mAP 50 improvement to YOLO-World. With scratch-trained detector weights and frozen vision-language encoders used only for prompt-side priors, YOLO-TVP achieves competitive prompt-conditioned detection while preserving real-time efficiency, verifying its advances in semantic alignment and prompt-adaptive detection.

VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval

Open-vocabulary object detection and segmentation aim to recognize arbitrary objects beyond predefined categories. Although recent vision-language and reference-based approaches have significantly advanced this field, they often rely on text prompts, limited visual examples, or expensive feature matching procedures, making them difficult to scale to large and continuously expanding object repositories. In this work, we propose VocaDet, a sample-driven open-vocabulary object detection and segmentation framework that learns object concepts directly from user-provided positive and negative sample collections without model retraining. The key idea is to transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database. Specifically, we employ DINOv3 as the visual feature extractor and apply agglomerative clustering with adaptive clustering sensitivity to generate multi-granularity visual tokens. These visual tokens, together with position-debiased representations and spatial topology information, are stored as expandable object memories in a vector database. During inference, query images are converted into visual tokens and efficiently matched against the stored object memories for object localization and segmentation. Furthermore, a background filtering mechanism is introduced to remove frequently occurring background patterns and reduce redundant retrieval operations in practical fixed-camera scenarios. Experiments on the UA-DETRAC dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training, while supporting continuously expandable recognition capability as additional positive and negative samples are accumulated.

Research on anti-forgetting training strategies for open-vocabulary object detection systems

Open-vocabulary object detection leverages vision-language pre-training to achieve cross-category generalization. With the progressive refinement of models such as GLIP, Grounding DINO, and YOLO-World, detectors have exhibited substantially enhanced zero-shot recognition capabilities on open vocabularies, with YOLO-World further advancing this capability into an efficient single-stage detection paradigm. However, during scenario-specific fine-tuning, training that relies solely on ground-truth annotations of base classes induces rapid degradation in the model’s discriminative ability for novel classes, often reverting to closed-set detector behavior. To address this issue, the present paper proposes a forgetting-resistant fine-tuning framework that decouples the training objectives into strong supervised adaptation for base classes and weak supervised preservation for novel classes. Specifically, novel pseudo-boxes are generated once by a frozen zero-shot teacher before any task-specific fine-tuning and are refined through threshold filtering, score weighting, and soft-ignore mechanisms to reduce the impact of noisy pseudo-supervision. Subsequently, teacher-guided logit distillation is incorporated, constraining the student model exclusively at the output distribution level to preserve its response relationships to the novel vocabulary. Experimental results demonstrate that conventional fine-tuning on YOLO-World-v2-S reduces Novel AP50 from 57.6 to 1.6. In contrast, through stepwise integration of the freezing strategy, pseudo-labels, and logit distillation, the proposed method elevates Novel AP50 to 51.6 and achieves an All AP50 of 53.7. These findings show that the proposed approach substantially mitigates catastrophic forgetting while preserving the original single-stage inference pipeline.

OVIP-SG: Open-Vocabulary Instance-Preserving Scene Graphs for Mapping and Retrieval of Small, Fine-Grained Objects

OVIP-SG is presented, a unified framework for instance-preserving semantic mapping, functional scene partitioning, and language-guided small, fine-grained object retrieval that outperforms ConceptGraphs under a unified evaluation protocol on Replica.

Text-guided zero-shot localization of unseen object categories