Skip to content

Advancing Video-Text Pretraining with Multi-View Captions

Sep 2026 · arXiv.org · Vol abs/2609.35090 · 0 citations · 50 references
Computer Science

TL;DR

This work proposes a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage, and introduces a granularity-aware text representation with separate CLS tokens for summary and detailed views.

Abstract

Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains underexplored. Existing web-scale datasets often provide only a single sparse caption per video that fails to capture rich spatiotemporal semantics, while directly using captioning models can generate noisy descriptions. We propose a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage. Starting from 10 million videos, our approach generates multi-view captions (MVC) through complementary summary and detailed captions, reasoning-based refinement, and semantic positive caption generation. To effectively exploit supervision at different granularities, we further introduce a granularity-aware text representation with separate CLS tokens for summary and detailed views. We pretrain video-text models using the resulting supervision corpus and evaluate them across standard, fine-grained and detailed text-to-video retrieval benchmarks. Our approach consistently improves both zero-shot and fine-tuned performance while using smaller pretraining corpora than existing methods, demonstrating the importance of rich and complementary textual supervision for video-text pretraining. Project page: https://rvandeghen.github.io/mvc/

View source

Similar papers

Sep 2026

OpenVid++: A Large-scale High-quality Dataset for Text-to-video Generation

A Motion-Aware Frame Sampling (MAFS) strategy for constructing motion-aware captions enables more effective utilization of video frames and improves captioning quality for large-motion videos.

Ke-Pan Nan, Tie-Han Fan, Rui Xie et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Query-Aware Streaming Latent Reasoning (QASLR), a post-training framework that accumulates evidence across clips while keeping the embedding size fixed, and support streaming latent aggregation as an effective approach to integrating long-video evidence into fixed-dimensional representations.

Hao-Zhe Chi, Song Jin, Yang Jin et al. · 1 citation
Conference Aug 2026

ActionLMM: captioning long-video actions with memory-augmented VLMs

This work introduces ActionLMM, a memory-augmented vision-language model for long-video action summarization that aligns visual and motion modalities through joint representation learning and leverages a novel dual-memory mechanism to retain both local motion details and global temporal structure.

Rui-Rui Li, Dari Abdullah Alrwoaily, Turgut Sofuyev et al. · 0 citations
#artificial intelligence Preprint Sep 2026

VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration

VidOmni-Bench is introduced, a benchmark that requires models to verify whether each event in dense video captions is supported by the video, and reveals three key findings: Video-LLMs frequently generate hallucinated descriptions in dense video captioning, and they also struggle as verifiers.

Changbeen Kim, Jun-Qing Chang, Kipyo Kim et al. · 0 citations
#machine learning Preprint Sep 2026

GleanVID: Complementary Token Selection for Efficient Video Large Language Models

Video Large Language Models (VideoLLMs) have achieved strong video understanding capabilities but incur substantial inference overhead due to the large number of visual tokens. Existing VideoLLM token compression methods largely rely on selection-independent scoring, overlooking cross-frame complementarity and conseque...

Shuo Yang, Chang-Bai Li, Rui Tang et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.