Semantic manifold-aware cross-modal similarity learning.

Abstract

Image-text matching (ITM) methods typically rely on positive-negative sample discrimination to learn cross-modal similarity. However, when similarity computation is primarily built upon sequence-level and linear representations, samples that are semantically similar but differ in underlying structural relations often remain ambiguously separated, resulting in blurred decision boundaries. To address this limitation, we propose a Semantic Manifold-Aware Similarity Learning (SMSL) framework, where the term "semantic manifold" is used in an operational sense to denote a topology-aware organization of textual semantic units induced from token/entity interactions, rather than a strict low-dimensional differentiable manifold. Specifically, the framework constructs a discrete semantic topology by disentangling intrinsic object, attribute, and relation dependencies within text, and injects the induced structural constraints back into the representation space through a topology injection mechanism, endowing textual embeddings with explicit topology awareness while preserving relational semantic continuity. The topology-aware textual representations are further exploited as semantic guidance to attend to and filter visual region features, reinforcing semantically relevant regions while suppressing redundant or distracting visual cues. After structure-aware enhancement on both the textual and visual sides, we introduce a dynamic threshold-based positive-negative decision boundary mechanism at the similarity computation stage. Unlike conventional fixed-margin strategies, this mechanism adaptively adjusts the decision boundary according to local semantic topology and cross-modal alignment certainty. In this way, the proposed method preserves the fundamental discriminative paradigm of ITM while shifting similarity evaluation from linear representation-level comparison to topology-aware similarity reasoning. Experiments on the Flickr30K and MS-COCO benchmarks demonstrate competitive performance in fine-grained ITM scenarios, validating the effectiveness of structurally informed and topology-aware similarity determination. The code is publicly available at: https://github.com/zhuliqi0309/SMSL.git.