This work introduces CANVAS (Contrastive Art-aware Network for Vision-Language Alignment with Sheaves), a framework for learning relation-aware multimodal representations inspired by sheaf theory, and evaluates on three newly introduced benchmarks of artworks for multi-relational art understanding.
Abstract
Understanding a painting is never a single act. Art historians may analyze the same work through concepts of style, iconography, or historical context, dimensions that are not interchangeable, and each carries distinct semantic relationships between the visual and the textual. Vision-Language Models (VLMs) like CLIP, which learn a single shared embedding space, collapse this richness into a single homogeneous alignment, thereby losing the multi-relational structure that defines art-historical reasoning. We introduce CANVAS (Contrastive Art-aware Network for Vision-Language Alignment with Sheaves), a framework for learning relation-aware multimodal representations inspired by sheaf theory. Each artwork is projected into multiple embeddings conditioned on the type of relation (i.e., the context), and a novel contrastive loss encodes contextual information during training, with no dependency on external data at inference. We evaluate on three newly introduced benchmarks of artworks for multi-relational art understanding: WikiArt+, derived from WikiArt and Wikipedia, HertzianaDP, from the Bibliotheca Hertziana collection, and SemArt+, refined from the SemArt dataset. In multimodal retrieval and art understanding, CANVAS outperforms the baselines, supporting the view that multi-relational alignment is not just theoretically motivated but also practically essential.
A group-supervised contrastive learning framework that addresses semantic ambiguity through two complementary mechanisms that expands parameter-based templates into diverse natural language descriptions using Large Language Models, and a group-supervised multi-positive contrastive objective that aligns text embeddings with sets of maps generated under shared control parameters is introduced.
Zhongyuan Xie, Kwanghee Won· International Conference on...· 0 citations
Recent vision-language models demonstrate impressive general visual understanding, yet their art interpretation remains shallow: they describe surface content but struggle with formal analysis, grounded historical interpretation, or affective characterization. We argue this is not only a model but also a dataset limitation. Existing art datasets are single perspective resources, where no dataset provides narrative, formal, emotional, and historical perspectives simultaneously for the same artworks. We introduce MMArt, a large-scale dataset of 74,234 WikiArt paintings, each annotated with four independently annotated perspectives plus a harmonized unified caption, produced by specialized vision-language models or human annotation and validated through complementary quality evaluations. Two complementarity analyses establish that perspectives encode genuinely distinct information. A generative analysis shows that formal analysis descriptions best preserve compositional style, and historical descriptions carry strong affective signal in reconstructed images. A discriminative retrieval analysis reveals task-asymmetry: narrative descriptions drive retrieval (R@1 = 44.0%), while formal descriptions, strongest for reconstruction, are nearly nondiscriminative at retrieval scale (R@1 = 7.8%). Leave-one-out analysis further confirms that historical descriptions are the least replaceable perspective across both tasks. Together, the two analyses establish that no single perspective suffices for all tasks, directly motivating MMArt multi-perspective design. The dataset, code, and additional information are available at https://shuaiwang97.github.io/MMArt/.
Shuai Wang, Wangyuan Ding, Yixian Shen et al.· 0 citations
Dense embeddings are foundational to contemporary natural language processing, information retrieval, recommendation, and retrieval-augmented generation. Nevertheless, a single general-purpose vector typically superimposes multiple relations—topic, entailment, sentiment, part–whole structure, evidential role, temporality, and domain-specific constraints—within one geometry and one similarity function. This article presents layered semantic refinement (LSR) as a framework and evaluation protocol rather than as a single universally validated algorithm. LSR retains a general encoder as a transferable semantic substrate while explicit modules reorganize, augment, or index its representations for a defined operational objective. The framework covers learned projections, nonlinear adapters, graph message passing, hierarchical aggregation, deterministic rule channels, clustering, multi-view representations, and prompt-conditioned embeddings.
Pedro Emílio Amador Salomão· Nexus Science Review· 0 citations
Museum and archival datasets do not mirror historical artistic production, but materialize the contingent histories of collecting, preservation, cataloging, and digitization. This has direct consequences for interpreting pretrained image representations: they may appear to encode historical time while actually encoding the institutional conditions under which objects become visible as data. We describe this phenomenon as temporal entanglement and investigate it by formulating artwork dating as an uncertainty-aware regression task over frozen image embeddings. We evaluate several pretrained vision models on a temporally controlled Wikidata corpus of artworks. Our results show that these models contain usable temporal information, with Vision-Language Models (VLMs) outperforming purely visual self-supervised baselines. However, a qualitative analysis indicates that this temporal knowledge is shaped by various biases.
Image-text matching (ITM) methods typically rely on positive-negative sample discrimination to learn cross-modal similarity. However, when similarity computation is primarily built upon sequence-level and linear representations, samples that are semantically similar but differ in underlying structural relations often remain ambiguously separated, resulting in blurred decision boundaries. To address this limitation, we propose a Semantic Manifold-Aware Similarity Learning (SMSL) framework, where the term "semantic manifold" is used in an operational sense to denote a topology-aware organization of textual semantic units induced from token/entity interactions, rather than a strict low-dimensional differentiable manifold. Specifically, the framework constructs a discrete semantic topology by disentangling intrinsic object, attribute, and relation dependencies within text, and injects the induced structural constraints back into the representation space through a topology injection mechanism, endowing textual embeddings with explicit topology awareness while preserving relational semantic continuity. The topology-aware textual representations are further exploited as semantic guidance to attend to and filter visual region features, reinforcing semantically relevant regions while suppressing redundant or distracting visual cues. After structure-aware enhancement on both the textual and visual sides, we introduce a dynamic threshold-based positive-negative decision boundary mechanism at the similarity computation stage. Unlike conventional fixed-margin strategies, this mechanism adaptively adjusts the decision boundary according to local semantic topology and cross-modal alignment certainty. In this way, the proposed method preserves the fundamental discriminative paradigm of ITM while shifting similarity evaluation from linear representation-level comparison to topology-aware similarity reasoning. Experiments on the Flickr30K and MS-COCO benchmarks demonstrate competitive performance in fine-grained ITM scenarios, validating the effectiveness of structurally informed and topology-aware similarity determination. The code is publicly available at: https://github.com/zhuliqi0309/SMSL.git.
Li-Qi Zhu, Dezhi Han, Chongqing Chen· Neural Networks· 0 citations
Leveraging textual information for image clustering has emerged as a promising direction, largely owing to the powerful representations learned by Vision-Language Models (VLMs). Existing approaches typically retrieve a textual counterpart for each image and then refine multimodal representations by directly enforcing cross-modal agreement, e.g., maximizing image-text similarity inherited from pretrained VLMs. However, such a strategy aligns heterogeneous representations across modalities without explicitly modeling the intrinsic structure within each modality and thus might yield unreliable alignment or distort modality-specific structures that are crucial for clustering. In this paper, we propose a simple but principled approach, termed deep modality-shared self-expressive model (DeepMORSE), which discovers cross-modal structures via a modality-shared self-expressive model and simultaneously learns structured representations that conform to a union of modality-specific subspaces. Moreover, we theoretically justify that the modality-shared self-expressive coefficients suppress inter-class noise towards a subspace-preserving solution, and show that mini-batch optimization procedure introduces an implicit regularization onto the self-expressive model. We evaluate our DeepMORSE on six widely used image clustering benchmarks and observe performance improvements exceeding 3% on the UCF-101, DTD-47, and ImageNet-Dogs datasets. In addition, we demonstrate the strong transferability of the learned representations by achieving state-of-the-art performance on downstream tasks such as image retrieval and zero-shot classification---without requiring any task-specific losses or post-processing. The code is available at: https://github.com/mengxianghan123/DeepMORSE.
Xianghan Meng, Wei He, Zhiyuan Huang et al.· 0 citations
