A novel Reliability-Aware framework for MMEA (RA-MMEA), which assesses visual reliability and adaptively improves unreliable visual representations for robust entity alignment and enables more reliable visual representations for modality fusion, thereby improving performance.
Abstract
The visual modality, i.e., images, plays a key role in multi-modal entity alignment (MMEA). Existing approaches often directly fuse the image with other modalities to align different entities. Although simple, such strategies overlook the potential noise in the images and their semantic misalignment with corresponding entities, resulting in suboptimal fusion and degraded performance. Addressing this, we propose a novel Reliability-Aware framework for MMEA (RA-MMEA), which assesses visual reliability and adaptively improves unreliable visual representations for robust entity alignment. The core lies in two modules, including dependency-aware visual reliability prediction (DA-VRP) and stability-regularized visual embedding generation (SR-VEG). The former aims to estimate the reliability of an image by leveraging multi-modal dependency within the entity, while the latter focuses on producing alternative visual representation conditioned on semantics encoded in textual modalities for multi-modal fusion. Compared to current methods, RA-MMEA enables more reliable visual representations for modality fusion, thereby improving performance. In extensive experiments, RA-MMEA achieves state-of-the-art results, verifying the importance of reliable visual modality for entity alignment and the effectiveness of RA-MMEA. The code and results will be released.
This approach introduces a Paired-Guided Consistency Rectification (PCR) module that utilizes strictly paired data as reliable anchors to dynamically calibrate the confidence of unpaired samples, effectively suppressing noise from inconsistent associations under mild-to-moderate corruptions.
Existing multi-modal image fusion (MMIF) methods generally face an optimization gap between the visual and semantic domains, limiting the enhancement of perception capabilities. To overcome this long-standing barrier, this work proposes W2F, a new visual-semantic mutual reinforcing paradigm for MMIF. First, we reinforc...
UAT-InfoNCE is proposed, a precision-weighted contrastive objective that adaptively modulates gradient contributions by the model’s own confidence, together with pairwise cross-modal disagreement terms that serve as zero-shot diagnostics for broadcast failure modes such as A/V desynchronisation, dubbed content, and adv...
Sanjib Kumar Swain, Santosh Kumar Swain· International journal of com...· 0 citations
Multi-modal image fusion (MMIF) aims to form a single image by integrating shared information, preserving complementary cues, and coordinating cross-modal conflicts across modalities. However, due to the absence of ground-truth fused images, existing MMIF supervision commonly uses spatial-domain sources or gradient var...
Ze-Yu Wang, Ming-Yu Ge, Hai-Yu Song et al.· arXiv.org· 0 citations
Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise va...
Jing-Hao Xu, Zhen-Hua Guo, Xiao-Feng Zhu et al.· 0 citations
Visual retrieval systems remain highly vulnerable to test-time distribution shifts caused by corruption, degradation, and environmental changes. Unlike classification, retrieval-time adaptation must preserve ranking consistency in a similarity space, where the dominant failure mode is not only inaccurate prediction con...
Yuan-Peng He, Li-Jian Li, Lin-Yu Li et al.· IEEE Transactions on Pattern...· 0 citations