Spherical Consistency and Test-time Adaptation for Open-Vocabulary Audio-Visual Event Localization
Abstract
Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) requires models to localize audio-visual events while predicting explicit categories for events unseen during training. Current baselines adopt a text-centered pairwise modality decision strategy. However, it lacks unified modeling of multimodal semantic consistency, rendering models more prone to errors when modalities are corrupted by noise. Meanwhile, existing methods suffer from severe open space risk, where models tend to misclassify unseen categories as seen ones during training. To mitigate these issues, we propose the Spherical Consistency and Test-time Adaptation (SCTA) framework. Unlike existing decision strategies, SCTA introduces the Circumscribed Hypersphere Alignment Representation Measure (CHARM) for multimodal consistency modeling. It uses the volume of the circumscribed hypersphere formed by modality vectors in high-dimensional space as a consistency measure, enabling a unified trimodal collaborative decision mechanism and improving noise robustness. To alleviate open space risk, we propose a forward decoupling and backward adaptation strategy. During forward propagation, we design a Subspace Decoupling Module (SDM) to enhance the discriminability of event semantics. In the backward phase, we pioneer integrating Test-Time Adaptation (TTA) into OV-AVEL by introducing Event Consistency TTA (ECTTA), which adjusts decision boundaries for each sample based on the test data. Extensive experiments on OV-AVEBench show that SCTA improves model capabilities in localization and generalization to unseen categories, significantly outperforming SOTA methods.