Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83\% average semantic accuracy across the four levels, compared with 37.28\% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.
Zhi Zeng, Cheng Zhang, Ze-Sheng Yang et al.· 0 citations
Audio-visual sound event localization and detection (AVSELD) seeks to identify and locate sound-emitting objects by leveraging both audio and visual data. Current methods primarily rely on convolutional neural networks (CNNs), whose constrained receptive fields limit their ability to capture broader contextual information. Although Transformer-based architectures exhibit considerable proficiency in capturing global contextual information, their efficacy is impeded by the quadratic computational complexity associated with processing long-range dependencies. This poses a significant bottleneck, particularly in scenarios involving longer sequence lengths. To overcome this limitation, we propose DSF-Net, a novel neural network that introduces a dual-strategy fusion approach for the AVSELD task. Built upon an efficient state-space model backbone to ensure linear complexity, DSF-Net is designed for robust and computationally efficient multi-modal comprehension. The proposed dual strategies consist of: (1) an Adaptive Frequency Fusion module that aligns and integrates features in the frequency domain, and (2) an Audio-aware Aggregation module that performs advanced feature integration while considering the consistency between modalities. These strategies are embedded within a progressive fusion framework to enhance overall feature learning. Extensive experiments on the STARSS2023 dataset validate our dual-strategy approach, demonstrating that DSF-Net achieves state-of-the-art performance and outperforms existing methods. The source codes are publicly available at https://github.com/Devin-Pi/avseld-mamba.
