Aug 2026· IEEE Transactions on Pattern Analysis and Machine Intelligence· Vol 48, pp. 15347-15364· 0 citations· 117 references
Medicine
TL;DR
An AVQA-specific two-stage framework that adapts established cross-modal reconstruction and dependency-modeling principles to the supervision constraints of TM-AVQA is developed, a setting in which modality-complete, audio-missing, and visual-missing samples may occur during both training and testing.
Abstract
Audio-Visual Question Answering (AVQA) methods typically assume both audio and visual modalities are available. In practice, however, an audio or visual stream may be unavailable because of passive signal loss, such as hardware or transmission failures, or intentional removal, such as withholding visual information for privacy. We formulate Training-time Modality-Missing AVQA (TM-AVQA), a setting where modality-complete, audio-missing, and visual-missing samples occur during both training and testing. To address this, we develop an AVQA-specific two-stage framework adapting cross-modal reconstruction to TM-AVQA constraints. In Stage-I, a reconstruction network infers missing-modality features from the available modality and question text. A Dense Temporal-scale Reconstruction (DTR) module aggregates multi-granularity information, while a Multimodal Dependency Modeling (MDM) module captures audio-visual-question dependencies. To supervise reconstruction without target features, we introduce Cross-Modal Relation-based Contrastive Learning (CMR-CL) to exploit multimodal relations, and Cross-Sample Relation-based Pseudo-label Learning (CSR-PL) to construct pseudo-targets from modality-complete samples. In Stage-II, the frozen reconstruction network combines with existing AVQA backbones for answer prediction. We construct controlled TM-AVQA variants of the MUSIC-AVQA, MUSIC-AVQA-R, and AVQA datasets. Experiments across multiple backbones, missing-modality conditions, and missing rates show robust improvements over existing baselines. Additional ablations validate the effectiveness of our task-specific designs, reconstruction quality, efficiency, and transferability to audio-visual recognition tasks.
SCoPE is introduced, a training-free framework in which all queried labels compete for shared evidence and each modality guides event selection in the other, and derives an exact condition for when this competition removes an FCA in a two-label fit.
J. Jeong, Junho Yoon, Hyunju Kim et al.· 0 citations
Open-vocabulary audio-visual event localization (OV-AVEL) grounds a text-queried event in time from video, audio, and language, with a reliability-aware asymmetric distillation framework.
Yi Xu, Cheng Chen, W. Lei· Proceedings of the 34th ACM...· 0 citations
A Low-Complexity Cross-Modal Alignment via Projection (LCAP) network is proposed, which introduces Projective Token Compression (PTC), which leverages Mish activation and adaptive average pooling to reduce feature redundancy while enhancing discriminative information, and Positional Spatial Enhancement (PSE), which exp...
Yu-Chen Sha, Lingli Wan, Ge Yang et al.· The Visual Computer· 0 citations
Audio Large Language Models (Audio LLMs) have advanced in audio understanding, yet they can still predict the answer by reasoning from textual cues or linguistic priors rather than the provided audio. A common remedy is to train models on data whose answers cannot be inferred from text alone. This approach can improve...
Hyebin Cho, Suho Yoo, Jihoo Jung et al.· 2 citations
ST-Omni-R1 is proposed, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning, and results on three public spatial-audio benchmarks indicate that its learned spatial and motion r...
Zhi Zeng, Cheng Zhang, Ze-Sheng Yang et al.· 2 citations
Video-HolmesV2 is presented, a novel benchmark designed for Deep Audio-Visual Coupling that enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence.
Zhao-Yang Wei, Zi-Peng Wang, Yu-She Cao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.