Skip to content
Open access

PWFAR: Patch–Word Fine-Grained Alignment for Long-Text Image–Text Retrieval

Aug 2026 · Electronics · Vol 15, pp. 3804 · 0 citations · 12 references

TL;DR

Results indicate that the proposed fine-grained alignment mechanism can effectively improve the model’s ability to capture local semantic relationships in long texts, thereby verifying the effectiveness of PWFAR for long-text image–text retrieval.

Abstract

The main challenge in long-text image–text retrieval lies in the fact that some existing vision–language models mainly rely on global image and text features for matching. Without explicit local alignment constraints, these models may struggle to fully capture complex semantic information in long textual descriptions, such as local objects, fine-grained attributes, and spatial relationships. To address the insufficient modeling of local details in global semantic matching, this paper proposes a patch–word fine-grained alignment retrieval model, named PWFAR. Built upon the Long-CLIP framework, the proposed method introduces an explicit fine-grained alignment mechanism between image patches and text tokens. By using textual words to guide the matching of local image regions, PWFAR enhances the model’s ability to capture local semantic correspondences. Specifically, PWFAR consists of three complementary training objectives: long-text global contrastive learning, short-text compact semantic supervision, and patch–word fine-grained alignment. These objectives are jointly optimized to constrain overall semantic consistency, core semantic stability, and local detail matching relationships. The model is trained on the 888k subset of the ShareGPT4V dataset and evaluated on the COCO2017 and Urban1k datasets. Experimental results show that PWFAR achieves competitive performance across different backbone networks and retrieval directions. Its clearest and most consistent gains are observed on the Urban1k long-text retrieval task, where it outperforms Long-CLIP and Long-CLIP-888k in both retrieval directions. These results indicate that the proposed fine-grained alignment mechanism can effectively improve the model’s ability to capture local semantic relationships in long texts, thereby verifying the effectiveness of PWFAR for long-text image–text retrieval.

Read PDF

Similar papers

Open access 2026

BiFGA: A Bidirectional Image–Text-Guided Fine-Grained Alignment Method for Remote Sensing Image–Text Retrieval

Remote sensing image–text retrieval matches remote sensing images with textual descriptions. However, dense objects, complex spatial layouts, and large-scale variations make fine-grained cross-modal alignment challenging. Existing methods mainly rely on global representations or direct patch-token similarities. Global matching may overlook small objects and spatial details, whereas local matching is susceptible to background clutter, scale variations, and redundant tokens. To address these limitations, this article proposes BiFGA, a bidirectional fine-grained alignment framework that uses cross-modal similarity responses as semantic anchors. Its bidirectionally guided fine-grained matching module reconstructs text-guided visual relations and vision-guided textual relations, enabling second-order interactions between image regions and textual units. BiFGA also introduces a hybrid window spatial-channel attention module to enhance visual regions and a channel-splitting phrase-level feature extraction module to model multiscale textual semantics. Global similarity and local fine-grained matching are integrated through coarse-to-fine scoring. Experiments on RSITMD, RSICD, UCM-Caption, and Sydney-Captions demonstrate the effectiveness and stability of BiFGA. Averaged over five independent runs, BiFGA achieves mR scores of 50.07%, 35.15%, 59.42%, and 53.19%, respectively. It obtains the best average mR among the compared methods on RSITMD, RSICD, and Sydney-Captions, while remaining comparable to the strongest method on UCM-Caption.

Yun Liao, Yong Liu, Junhui Liu et al. · 0 citations
Preprint Aug 2026

A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval

It is shown that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens, and paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning.

Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz et al. · 0 citations
Aug 2026

PHA-Net: Prototype-based hierarchical alignment network for text-video retrieval

A new prototype-based hierarchical alignment network (PHA-Net) to align individual/local/global level representations across modalities and introduces multiple modality-shared prototypes as the bridge to efficiently optimize text and video representations for cross-modal alignment.

Xiaolun Jing, Kezhao Yin, Xinxing Yang et al. · 0 citations
Open access Jul 2026

Q-GrAM: Fine-Grained Image–Text Retrieval via Grouped Query Routing and Conditional Query Modulation

Existing image–text retrieval methods often compute cross-modal similarity using global single-vector representations. Although efficient for coarse semantic alignment, such compressed representations are limited when textual queries involve fine-grained semantics, including objects, attributes, relations, and their compositional structures. This paper focuses on fine-grained text-to-image retrieval and proposes Q-GrAM, a retrieval-oriented adaptation of the BLIP-2 Q-Former. Instead of treating Q-Former queries as a homogeneous set, Q-GrAM partitions a fixed query budget into semantically differentiated groups. A text-guided router assigns token-level semantic demands to query groups, while query conditional initialization modulates each group according to group-level textual summaries. The resulting grouped visual query features are matched with text tokens through a group-aware late interaction scorer, and auxiliary routing balance and inter-group diversity regularization are introduced to stabilize semantic specialization. Experiments on MS-COCO 5K, Flickr30K, and Flickr30K-CFQ show that Q-GrAM achieves strong text-to-image retrieval performance against both global embedding baselines and representative fine-grained image–text matching methods, while maintaining competitive bidirectional retrieval performance. These results demonstrate the effectiveness of structured, text-conditioned Q-Former query specialization for fine-grained text-driven image search.

Guihe Gu, Huawei Li, Hong Qin · 0 citations
Conference Aug 2026

GLCE: global-local channel enhancer for fine-grained e-commerce image-text retrieval

Vision-language pre-trained (VLP) models excel in general cross-modal image-text retrieval. However, when fine-tuned for fine-grained e-commerce retrieval, the standard Vision Transformer backbone suffers from channel information equality: it weights all feature channels equally and fails to emphasize critical fine-grained visual cues (e.g., brand logos, materials, textures). This defect reduces cross-modal discriminability, increases hard negative confusion, and limits retrieval accuracy. To solve this problem, we propose GLCE (Global-Local Channel Enhancer), a lightweight plug-and-play module with a dual-branch structure. The global branch captures long-range channel semantic co-occurrence through a strict information bottleneck, while the local branch models fine-grained channel interactions via dimension-preserving 1D convolution. When integrated with full fine-tuning, GLCE breaks the performance bottleneck of strong baselines with nearly no extra parameters. Extensive experiments on the MUGE dataset show that our method achieves comprehensive improvements, with a mean recall of 75.90% and a notable boost in top-5 recall. This work validates the effectiveness of fine-grained channel recalibration for VLP domain adaptation, provides a lightweight industrial solution for fine-grained cross-modal retrieval, and offers new insights for channel attention design in Transformer-based vision-language architectures.

Qixuan Pan · 0 citations