MODAL is a novel multi-modal object re-identification framework, grounded in coupled sparse coding theory and differential suppression principles, that achieves state-of-the-art performance with superior transparency.
Abstract
Multi-modal object re-identification (Re-ID) aims to facilitate cross-camera object retrieval in complex environments by leveraging complementary information from visual (e.g., RGB, NIR, TIR) and textual modalities. However, existing approaches often lack principled feature disentanglement and coherent multi-modal integration, leading to entangled representations that introduce cross-modal conflicts, obscure discriminative cues, and suffer distribution shift under modality-missing conditions. To tackle these challenges, we propose MODAL, a novel multi-modal object re-identification framework, grounded in coupled sparse coding theory and differential suppression principles. A core component of MODAL is a Multi-modal Feature Sparse Decoupling module, developed in a model-driven deep unrolling manner based on multi-modal coupled sparse coding. It explicitly decomposes multi-modal features into uni-modal specific, bi-modal and tri-modal shared representations, thereby achieving more transparent and effective feature disentanglement. Benefiting from the principled feature disentanglement, MODAL naturally mitigates performance degradation in incomplete-modality scenarios via a Modality-Aware Subspace Activation that selectively activates only the consistently shared subspaces. Moreover, we propose a Text-Image Differential Filtering module that leverages coarse-grained textual semantics to adaptively suppress task-irrelevant responses in the decoupled visual representations, thereby enhancing discriminative information. Extensive experiments on four datasets demonstrate that MODAL achieves state-of-the-art performance with superior transparency.
A unified multi-modal tracker Diff-MM is proposed by exploiting the multi-modal understanding capability of the pre-trained text-to-image generation model by harnessing the extensive prior knowledge in the generation model to achieve a unified tracker with uniform parameters for RGB-N/D/T/E tracking.
Shi-Yu Xuan, Ze-Chao Li, Jin-Hui Tang et al.· IEEE Transactions on Pattern...· 0 citations
Multi-modal object re-identification exploits complementary RGB, near-infrared (NIR), and thermal-infrared (TIR) observations to retrieve target objects. However, existing methods commonly employ visual encoders optimized for global image-text alignment and select tokens using learned importance scores. Such designs fa...
Wan-Ke Xia, Rui Zhu, Xing-Guo Xu et al.· 0 citations
The rapid spread of multimodal misinformation on social media platforms has strengthened the need for effective fake news detection systems. While recent multimodal approaches integrate textual and visual information, most existing studies formulate the task as binary classification, thereby failing to fully exploit th...
Idza Aisara Norabid, Masita Jalil, Noor Ashikin Abd Rahim· International Journal of Adv...· 0 citations
This work proposes a novel framework centered on Positive-Incentive Noise and structured prompt modulation that harnesses task-aware {\pi}-noise, sampled from a distribution conditioned on both visual and text inputs, to perturb global tokens and enable semantics-guided cross-modal compensation.
Wei-Xiang Zhou, Yu-Hao Wang, Xing-Guo Xu et al.· 0 citations
As a fundamental task in multimedia information retrieval, cross-modal image-text retrieval connects heterogeneous visual and linguistic modalities to support bi-directional image and text search by creating semantic correlations. While foundational models represented by CLIP provide high-quality baseline features and...
Min Lin, Shu-Hua Zhu, Jia-Hao Zhang· Conference on Advanced Algor...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.