Skip to content

Generative Uncertainty as a Self-supervised Signal for Semantic Similarity Learning

Sep 2026 · arXiv.org · Vol abs/2609.35341 · 0 citations · 37 references
Computer Science

TL;DR

A novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA.

Abstract

Evaluating semantic similarity between videos is a fundamental challenge in computer vision, essential for tasks ranging from out-of-distribution (OOD) detection to video retrieval. However, defining and labeling video similarity is notoriously difficult and expensive due to the complex spatio-temporal nature. In this paper, we propose a novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations. Our method is based on the observation that T2V models produce consistent outputs for familiar concepts but exhibit high variance and uncertainty when prompted with specialized concepts. We utilize this behavior to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA. Specifically, we learn a mask over these embeddings using purely generated data, encouraging the model to retain features that remain consistent across generations of general concepts while discarding those associated with generative noise or uncertainty. Experimental results across three key tasks demonstrate that our learned feature subspaces consistently outperform original pretrained features and baseline feature selection methods.

View source

Similar papers

Aug 2026

Semantic manifold-aware cross-modal similarity learning.

A Semantic Manifold-Aware Similarity Learning (SMSL) framework, where the term "semantic manifold" is used in an operational sense to denote a topology-aware organization of textual semantic units induced from token/entity interactions, rather than a strict low-dimensional differentiable manifold.

Li-Qi Zhu, De-Zhi Han, Chong-Qing Chen · 0 citations

with Pretrained Generative Model for Self-Supervised Learning

GenView is presented, a controllable framework that augments the diversity of positive views leveraging the power of pretrained generative models while preserving semantics, and an adaptive view generation method that dynamically adjusts the noise level in sampling to ensure the preservation of essential semantic meani...

Xiao-Jie Li, Yi-Bo Yang, Xiang-Tai Li et al. · 0 citations
Conference Open access Sep 2026

Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Interestingly, despite using zero manual annotations, MoSA achieves segmentation performance comparable to the fully supervised SAM, revealing that harnessing large-scale unlabeled motion is a feasible and highly scalable alternative to annotation-driven segment-anything pipelines.

Wei-Jian Jian, Xiao-Yue Zhang, Bin Xiao et al. · 0 citations
Aug 2026

Self-supervised skeleton action recognition based on graph prototype learning

This work presents a novel self-supervised architecture centered on graph prototype learning that sets a new state-of-the-art on the ARMM dataset with an accuracy of 95.70%, substantiating the efficacy and transferability of prototype-guided self-supervised learning for skeleton-based action representation.

Zhijie Xu, Hong-Wei Chen, Xia Li · 0 citations
Preprint Aug 2026

G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification

G2D is proposed, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image and transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning.

Zehua Hao, Fang Liu, Qinliang Wang et al. · 0 citations
Sep 2026

Multimodal graph-based fusion via image descriptions for few-shot open-set recognition.

A multimodal Graph-based Fusion (MGF) framework that learns visually grounded semantic representations to enhance FSOR performance and achieves superior open-set recognition and competitive closed-set classification performance.

Xilang Huang, Seon-Han Choi · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.