DiCE-CIR is proposed, a direct composition learning method that predicts composed query representations by directly composing a reference image and an edit text and achieves state-of-the-art performance on CIRCO and competitive performance on CIRR while maintaining high computational efficiency.
Abstract
Zero-shot composed image retrieval (ZS-CIR) aims to retrieve a target image from a multimodal query consisting of a reference image and an edit text describing the desired modification. Recent ZS-CIR studies have relied on projection-based methods that map a reference image into pseudo-word tokens in the text embedding space. However, such methods require additional projection and re-encoding steps, increasing training complexity, reducing efficiency, and introducing a discrepancy between training and inference. In this paper, we propose DiCE-CIR, a direct composition learning method that predicts composed query representations by directly composing a reference image and an edit text. To enable scalable training without manually annotated triplets, we automatically construct compositional training samples from large-scale image-caption pairs using a large language model. Based on these samples, we train a lightweight composition module with objectives that promote alignment with the target, edit-consistent semantic transformation, and retrieval discriminability. We conduct extensive experiments on ZS-CIR benchmarks and show that DiCE-CIR achieves state-of-the-art performance on CIRCO and competitive performance on CIRR while maintaining high computational efficiency.
This paper introduces a Vision-Free CIR framework that addresses this challenge through two key techniques: Attribute-Augmented Hybrid Scoring, which compensates for lost visual details via explicit attribute matching, and LLM-Based Reranking, which verifies semantic consistency of top candidates.
Ryotaro Shimada, Yu-Chieh Lin, Yuji Nozawa et al.· 1 citation
A new model based on a Large Multimodal Model (LMM) that functions as a context-aware reasoner for CoCo-IR is proposed, which interprets the entire interaction history to generate Transformable Image Embeddings (TIE) that evolve across turns.
Shengcao Cao, T. Dabral, Z. Ding et al.· 0 citations
Training-free zero-shot composed image retrieval finds a target image in a gallery from a reference image and a text edit without learning from task-specific image triplets. Existing methods typically describe the target as a whole and match this description with a global image representation. This global matching can mix different semantic cues and lose fine- grained details. We propose MULVEC, a role-aware method whose compiler produces a structured query record that is mapped to four retrieval roles: Global describes the full target, Desired states what should appear, Preserve states what should remain, and Forbidden states what should disappear. Frozen encoders map the query to one target description vector and role-specific probe vectors, while each candidate is represented by one global visual vector and a bank of local visual vectors. The retrieval roles then use this shared evidence for their respective purposes, and a fixed weighted sum of their scores ranks the entire gallery in a single retrieval pass. Across CIRCO, CIRR, and FashionIQ and three backbone scales, MULVEC improves CIRCO mAP@5 by up to 23.0% over the strongest compared method and gives the best CIRR and FashionIQ results in our comparison.
Zihao Zhang, Dayan Wu, Xinze Liu et al.· 0 citations
ReCap is a plug-and-play framework that shifts synthetic data refinement from implicit global matching to explicit fine-grained realignment, and enforces entity-level correspondence by using detected image-supported entities to guide caption rewriting, yielding more faithful synthetic supervision.
Zhiyue Liu, Wenkai Zhou, Jian Qin et al.· 0 citations
It is shown that paragraph supervision enables effective use of long token sequences, whereas caption-only training degrades beyond 60 tokens, and paragraph supervision consistently benefits long-description benchmarks and hard negatives prove detrimental in text-only fine-tuning.
Mahyar Ghazanfari, Amin Tabrizian, Arsyi Aziz et al.· 0 citations
Given a reference image, composed image retrieval aims to search for the target image from the image gallery with a modification sentence. The complicated semantic relationships within the composed query to comprehend the preserved visual context and modified semantics pose a challenge for the researchers. Most current approaches concentrate on the global or fragmental composition to combine the multi-modal query features and bridge the distance between the query and the target. However, global-level matching tends to ignore fine-grained components that distinguish the matched instance from hard negative instances. As for the fragment-level matching, the over-sensitivity to the local noise due to the lack of explicit local labels and excessive computational demands may have a detrimental impact on performance. In this paper, we propose Semantic Composition via Optimal Transport (SCOT) for composed image retrieval to address the aforementioned issues. To optimize the computational resources, SCOT first introduces Hybrid-modal Affinity Summarization (HAS) to compact salient relationships in the affinity matrix from the reference image patches and modification tokens. Furthermore, to mitigate the over-confidence in local noise, SCOT adaptively assigns optimal transport weights on the salient aligned fragments to Retain Prototype from Alignments (RAS) and guides discriminant semantics learning to Update Semantics from Enhanced Embeddings (USEE), while preserving the characteristics from global structures. Specifically, RAS module investigates the preserved semantics from the reference image based on the affinity matrix, meanwhile USEE module distills significant modification semantics. The proposed SCOT has demonstrated advanced experimental performances on widely-used datasets FashionIQ, CIRR, and Fashion200K. Ablative studies manifest the effectiveness of optimal transport guidance on preserved visual prototypes and updated semantics.
Yifan Wang, Wuliang Huang, Chun Yuan· IEEE Transactions on Image P...· 0 citations