Skip to content

SyncRA: Learning Temporal Correspondence in Omni-Modal Models

Sep 2026 · 0 citations · 25 references
Computer Science

TL;DR

The proposed Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision, consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations.

Abstract

Recent omni-modal models demonstrate strong perception of audio and visual inputs, yet often struggle to connect what they hear with what they see at the same moment. This weakness in temporal correspondence can cause models to associate spoken cues with the wrong visual scenes, producing plausible answers grounded in incorrect audio-visual pairings. We diagnose this problem through controlled temporal swaps, revealing that model answers do not reliably follow changes in these pairings. To address it, we propose Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision. Specifically, SyncRA contrasts intermediate audio-visual representations within each video, aligning matching moments while separating mismatched ones to capture local temporal correspondence within a shared global context. The objective derives supervision directly from existing input timing, requiring no additional annotations and leaving inference unchanged. We evaluate SyncRA across four open omni-modal models spanning different sizes and architectures on five public video benchmarks. SyncRA consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations. These results demonstrate that lightweight, targeted supervision can effectively strengthen temporal correspondence and translate into broad improvements in audio-visual question answering.

View source

Similar papers

Preprint Sep 2026

Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?

Video-HolmesV2 is presented, a novel benchmark designed for Deep Audio-Visual Coupling that enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence.

Zhao-Yang Wei, Zi-Peng Wang, Yu-She Cao et al. · 0 citations
Preprint Oct 2026

TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding

TiTok is presented, an audio-visual large language model (AV-LLM) that localizes an arbitrary number of temporal event segments for each query, and the Time Token Interleaving (TTI) method is introduced, which explicitly injects special time tokens into the audio-visual stream to align input-side temporal perception wi...

Eunji Shin, Dahyun Choi, Seungyeon Jo et al. · 0 citations
#machine learning Preprint Oct 2026

LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes...

B. Robson, Santeri Mentu, Wen-Shuai Zhao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models

Findings show that semantic reference-text overlap should not be treated as a proxy for temporal localization, and that AVTrace can identify task-specific weaknesses while providing a testbed for temporal post-training.

Long-Yin Zhang, P. Mahendra, Chengwei Wei et al. · 1 citation

Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

DMC-Repair is proposed, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question, and reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-ques...

Yue-Ran Ma, Ronghao Lin · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.