Skip to content
Log in
Open access

RoFLIP: Robust and Fine-Grained Alignment for Vision-Language Compositional Reasoning

Yiwei Sun Chuanbin Liu Shancheng Fang Pandeng Li Zhiying Lu Hongtao Xie
Aug 2026 · International Journal of Computer Vision · Vol 134 · 0 citations · 100 references

TL;DR

The Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP) is proposed, enhancing both the robustness and granularity of vision-language alignment and underscore RoFLIP’s compositional reasoning and generalization abilities.

Abstract

Vision-language models pretrained on large-scale datasets have become the foundation of cross-modal understanding. However, they often overemphasize salient objects and underperform in compositional reasoning that additionally requires fine-grained perception of attributes and inter-object relations. Although compositional data-driven approaches have shown promise in addressing this challenge, low-quality captions undermine robust alignment and consequently hinder fine-grained perception. Specifically, two fundamental issues arise: (1) low-density positive captions restrict the global representation capability of image encoders, causing local alignment. (2) false-negative captions introduce noise into training, resulting in fragile alignment. In response, we propose the Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP), enhancing both the robustness and granularity of vision-language alignment. To address the first challenge, RoFLIP employs the Densified Ranking Loss (DRL), which promotes global visual representations by contrasting dense and sparse captions. For the second challenge, RoFLIP incorporates Denoised Cross-modal and Text-modal Contrastive Losses (DCCL &DTCL), reducing the impact of noise while effectively leveraging negative captions. In addition, we introduce RoFLIP’s Auxiliary dataset (RoFA), comprising 225K high-quality captions that are fluent and image-related to support training. Quantitative evaluations demonstrate that RoFLIP-CLIP-B/32 achieves an average gain of 7.41%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7.41\%$$\end{document} across 5 compositional reasoning benchmarks while maintaining considerable performance on 23 downstream benchmarks. Furthermore, replacing the corresponding modules in LLaVA-v1.5 and Stable Diffusion v1.5 with RoFLIP-CLIP-L/14 also brings performance gains on VQA and image generation tasks, respectively. These results underscore RoFLIP’s compositional reasoning and generalization abilities.

DOI Semantic Scholar
Read PDF

Similar papers

Preprint Jul 2026

U-shaped Multi-granularity Learning for Vision-Language Models

The prompt learning paradigm for vision-language models is effective yet faces a granularity dilemma: global prompts lack fine-grained semantic awareness, while local prompts ignore contextual associations, limiting cross-task generalization. This dilemma exists in dense prediction tasks. Inspired by U-Net, which unifies multi-level representations across granularities, we propose UPrompt, a U-shaped multi-granularity prompt learning framework for vision-language models. Similar to how U-Net integrates fine and coarse features through symmetric encoder-decoder pathways with cross-level connections, UPrompt constructs parallel multi-granularity representations in both visual and textual modalities, where coarse-to-fine cascaded enhancement propagates global context to refine local details, while fine-to-coarse hierarchical supervision ensures semantic consistency across scales. Extensive experiments on 17 benchmarks validate our effectiveness. UPrompt outperforms MAMET and VPKE by 4.1 and 7.3 rSum on MSCOCO, surpasses CoCoA-Mix by 5.09% in base-to-novel generalization, while maintaining competitive performance with minimal overhead (coarse-grained) and matching PSRC with 1/3 cost (medium-grained).

Biao Chen, Yunqian Yu, Xiang-Xu Zhao et al. · 0 citations
Preprint Jul 2026

MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

This work introduces \method, a Multi-scale Adaptive Vision Encoder, a Multi-scale Adaptive Vision Encoder that uses position-dependent gates to fuse shallow, intermediate, and deep features from a vision Transformer, preserving global semantics while enhancing edges, text, and local structure.

Sha Lei · 0 citations
Preprint Jul 2026

CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models

Cross-image comparative reasoning remains challenging for vision-language models (VLMs), especially when correct prediction requires fine-grained attribute grounding and globally consistent reasoning. We present CoRe, a unified framework for this problem. CoRe includes: (i) CoRe-20K, a large-scale triplet-based training set automatically constructed from structured visual metadata through a multi-expert collaborative pipeline, covering counting, depth, distance, and spatial relations; (ii) TriSR, a structured reward framework that jointly supervises attribute grounding, judgment alignment, and triplet consistency under GRPO optimization; and (iii) CoRe-Bench, the first benchmark dedicated to fine-grained cross-image comparative reasoning. Experiments show that CoRe substantially outperforms existing VLMs on CoRe-Bench while remaining competitive on standard multimodal benchmarks, achieving a 28.2-point gain in partial accuracy over the strongest baseline.

Lin Peng, Cong Wan, Zeyu Guo et al. · 0 citations
Preprint Aug 2026

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.

Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao et al. · 0 citations