Sep 2026· ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)· 0 citations· 60 references
TL;DR
This study designs a Text-Aware Patch Selection module, which dynamically constructs patch-level regions for fine-grained excavation based on informative token selection, effectively bypassing the need for bounding boxes, and introduces a Cross-Grained Alignment module to learn modality-shared features across global and object levels.
Abstract
Vision-Language Pre-training (VLP) aims to convert vision-language tasks into image-text matching challenges, necessitating robust interaction between visual and textual modalities. However, current research often faces a trade-off: methods pursuing fine-grained alignment typically rely on computationally expensive object detectors and pre-defined annotations, while coarse-grained alignment methods overlook subtle visual details. To alleviate these issues, we present OMEGA, a COmprehensive Multi-lEvel Granularity Alignment framework. OMEGA achieves sufficient cross-modal alignment without relying on external object detectors or annotations. Specifically, we first design a Text-Aware Patch Selection (TAPS) module, which dynamically constructs patch-level regions for fine-grained excavation based on informative token selection, effectively bypassing the need for bounding boxes. Secondly, to facilitate multi-grained information fusion, we introduce a Cross-Grained Alignment (CGA) module to learn modality-shared features across global and object levels. Experimental results demonstrate that OMEGA surpasses existing approaches in five pivotal vision-language tasks—image-text retrieval, visual question answering, visual reasoning, visual entailment, and image captioning, while maintaining superior inference efficiency compared to detector-based methods. We believe this study offers a promising object-free paradigm for scalable vision-language pre-training.
The Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP) is proposed, enhancing both the robustness and granularity of vision-language alignment and underscore RoFLIP’s compositional reasoning and generalization abilities.
Yiwei Sun, Chuan-Bin Liu, Shan-Cheng Fang et al.· International Journal of Com...· 0 citations
The Granularity-Aware Region Alignment and Semantic Prototype (GRASP) learning framework is proposed, enhancing discriminative capability through two synergistic strategies, and Semantic Perturbation Enhanced Matching (SPEM), which leverages a foreground-purified Semantic Prototype Codebook to construct semantically pe...
Jia-Hui Cui, Yan Zhao, Kang-Ye Wei et al.· 0 citations
A Low-Complexity Cross-Modal Alignment via Projection (LCAP) network is proposed, which introduces Projective Token Compression (PTC), which leverages Mish activation and adaptive average pooling to reduce feature redundancy while enhancing discriminative information, and Positional Spatial Enhancement (PSE), which exp...
Yu-Chen Sha, Lingli Wan, Ge Yang et al.· The Visual Computer· 0 citations
Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding.
Changhao Xiang, Shangyu Xing, Zhen Wu et al.· 0 citations
MODdapter extracts visual semantic cues from coarse segmentation results and integrates them as supplementary textual data, allowing for a more accurate projection of text features into the feature space and enhancing fine-grained recognition of unseen objects.
Jia-Xiang Fang, Shi-Qiang Ma, Jing Wang et al.· Neural Networks· 0 citations
The proposed Cross-Modal Adaptive Token Selection and Alignment Network (CATSANet), a CLIP-based framework tailored for TI-ReID, achieves competitive performance in terms of Rank-k accuracy and mAP, demonstrating the effectiveness of fine-grained alignment and ranking refinement across datasets.
Dongbin Chen, Junjie Li, Hao Xu et al.· Pattern Analysis and Applica...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.