Collaborative Multimodal Entity Linking via Multi-Channel Neural Cross-Modal Interaction with Synergistic Consistency Optimization

Abstract

Multimodal entity linking (MEL) grounds entity mentions in text-image contexts to entries in a structured knowledge base; however, most existing systems still decompose a multimodal document into independent mention-level decisions. This formulation overlooks a central tension of real-world MEL: the evidence needed to disambiguate an ambiguous mention is often distributed across co-occurring mentions, visual context, and cross-modal consistency, rather than being contained in the mention itself. To address this limitation, we propose Collaborative Entity Linking through Multichannel Interaction (CELMI), a four-channel framework that jointly models textual semantics, visual perception, cross-modal alignment, and inter-mention collaboration. CELMI employs dual-level textual alignment, text-guided visual gating, learned cross-modal projection, and attention-based entity graph propagation. To stabilize joint optimization, we further introduce a multi-channel consistency objective that combines per-channel contrastive losses with an overall ranking loss, reducing channel dominance and representation collapse. Among conventional non-LLM/VLM MEL models, CELMI achieves the strongest MRR and Hits@1 performance on WikiMEL, RichpediaMEL, and WikiDiverse, with Hits@1 scores of 89.03% on WikiMEL and 83.02% on RichpediaMEL; LLM/VLM systems remain stronger on WikiDiverse, positioning CELMI as a lightweight complement. Progressive stress ablation shows a 38.50 percentage-point absolute Hits@1 drop when the architecture is reduced to a single unimodal endpoint, confirming that the gains arise from synergistic channel interaction rather than isolated module effects.