The proposed Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision, consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations.
Abstract
Recent omni-modal models demonstrate strong perception of audio and visual inputs, yet often struggle to connect what they hear with what they see at the same moment. This weakness in temporal correspondence can cause models to associate spoken cues with the wrong visual scenes, producing plausible answers grounded in incorrect audio-visual pairings. We diagnose this problem through controlled temporal swaps, revealing that model answers do not reliably follow changes in these pairings. To address it, we propose Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision. Specifically, SyncRA contrasts intermediate audio-visual representations within each video, aligning matching moments while separating mismatched ones to capture local temporal correspondence within a shared global context. The objective derives supervision directly from existing input timing, requiring no additional annotations and leaving inference unchanged. We evaluate SyncRA across four open omni-modal models spanning different sizes and architectures on five public video benchmarks. SyncRA consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations. These results demonstrate that lightweight, targeted supervision can effectively strengthen temporal correspondence and translate into broad improvements in audio-visual question answering.
Video-HolmesV2 is presented, a novel benchmark designed for Deep Audio-Visual Coupling that enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence.
Zhao-Yang Wei, Zi-Peng Wang, Yu-She Cao et al.· 0 citations
TiTok is presented, an audio-visual large language model (AV-LLM) that localizes an arbitrary number of temporal event segments for each query, and the Time Token Interleaving (TTI) method is introduced, which explicitly injects special time tokens into the audio-visual stream to align input-side temporal perception wi...
Eunji Shin, Dahyun Choi, Seungyeon Jo et al.· 0 citations
OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework, is introduced, providing a fine-grained roadmap for omnimodal development.
Zhongyu Yang, Jia-Le Tao, Rui-Tao Chen et al.· 1 citation
Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes...
B. Robson, Santeri Mentu, Wen-Shuai Zhao et al.· 0 citations
Findings show that semantic reference-text overlap should not be treated as a proxy for temporal localization, and that AVTrace can identify task-specific weaknesses while providing a testbed for temporal post-training.
Long-Yin Zhang, P. Mahendra, Chengwei Wei et al.· 1 citation
DMC-Repair is proposed, which trains models on the same kind of cross-modal swapped samples while assigning supervision according to the modality specified by the question, and reduces the image-induced share of the answer effect by 59.9%, effectively suppressing the cross-modal shortcut without compromising audio-ques...
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.