Skip to content
Open access

Cross-Modal Comprehensive Multi-Level Granularity Alignment for Vision-Language Pre-Training

Sep 2026 · ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP) · 0 citations · 60 references

TL;DR

This study designs a Text-Aware Patch Selection module, which dynamically constructs patch-level regions for fine-grained excavation based on informative token selection, effectively bypassing the need for bounding boxes, and introduces a Cross-Grained Alignment module to learn modality-shared features across global and object levels.

Abstract

Vision-Language Pre-training (VLP) aims to convert vision-language tasks into image-text matching challenges, necessitating robust interaction between visual and textual modalities. However, current research often faces a trade-off: methods pursuing fine-grained alignment typically rely on computationally expensive object detectors and pre-defined annotations, while coarse-grained alignment methods overlook subtle visual details. To alleviate these issues, we present OMEGA, a COmprehensive Multi-lEvel Granularity Alignment framework. OMEGA achieves sufficient cross-modal alignment without relying on external object detectors or annotations. Specifically, we first design a Text-Aware Patch Selection (TAPS) module, which dynamically constructs patch-level regions for fine-grained excavation based on informative token selection, effectively bypassing the need for bounding boxes. Secondly, to facilitate multi-grained information fusion, we introduce a Cross-Grained Alignment (CGA) module to learn modality-shared features across global and object levels. Experimental results demonstrate that OMEGA surpasses existing approaches in five pivotal vision-language tasks—image-text retrieval, visual question answering, visual reasoning, visual entailment, and image captioning, while maintaining superior inference efficiency compared to detector-based methods. We believe this study offers a promising object-free paradigm for scalable vision-language pre-training.

Read PDF

Similar papers

Open access Aug 2026

RoFLIP: Robust and Fine-Grained Alignment for Vision-Language Compositional Reasoning

The Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP) is proposed, enhancing both the robustness and granularity of vision-language alignment and underscore RoFLIP’s compositional reasoning and generalization abilities.

Yiwei Sun, Chuan-Bin Liu, Shan-Cheng Fang et al. · 0 citations
Preprint Aug 2026

GRASP: Granularity-Aware Region Alignment and Semantic Prototype Learning for Fine-Grained Cross-Modal Understanding in Drone Views

The Granularity-Aware Region Alignment and Semantic Prototype (GRASP) learning framework is proposed, enhancing discriminative capability through two synergistic strategies, and Semantic Perturbation Enhanced Matching (SPEM), which leverages a foreground-purified Semantic Prototype Codebook to construct semantically pe...

Jia-Hui Cui, Yan Zhao, Kang-Ye Wei et al. · 0 citations
Aug 2026

Cross-modal alignment enhancement for lightweight large vision language models

A Low-Complexity Cross-Modal Alignment via Projection (LCAP) network is proposed, which introduces Projective Token Compression (PTC), which leverages Mish activation and adaptive average pooling to reduce feature redundancy while enhancing discriminative information, and Positional Spatial Enhancement (PSE), which exp...

Yu-Chen Sha, Lingli Wan, Ge Yang et al. · 0 citations
Sep 2026

MODdapter: Spatially-aware text embeddings for zero-shot semantic segmentation.

MODdapter extracts visual semantic cues from coarse segmentation results and integrates them as supplementary textual data, allowing for a more accurate projection of text features into the feature space and enhancing fine-grained recognition of unseen objects.

Jia-Xiang Fang, Shi-Qiang Ma, Jing Wang et al. · 0 citations
Aug 2026

CATSANet: cross-modal semantic token selection and optimal transport based part alignment for text-to-image person ReID

The proposed Cross-Modal Adaptive Token Selection and Alignment Network (CATSANet), a CLIP-based framework tailored for TI-ReID, achieves competitive performance in terms of Rank-k accuracy and mAP, demonstrating the effectiveness of fine-grained alignment and ranking refinement across datasets.

Dongbin Chen, Junjie Li, Hao Xu et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.