This paper proposes SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams and achieves state-of-the-art performance on the seen and unseen splits.
Abstract
Open-vocabulary audio-visual event localization aims to infer explicit segment-level categories for both seen and unseen audio-visual events. Existing solutions rely heavily on hand-crafted prompt templates, whose predictions are highly sensitive to subtle wording variations. Learnable prompts remove this manual tuning burden, but when trained solely on seen events, they tend to overfit to seen categories and distort the inter-event semantic topology required for generalization to unseen categories, a failure we term semantic topology drift. In this paper, we propose SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams. Specifically, our SIMPLE combines Learnable Continuous Prompt Tuning (LCPT) with Semantic Topology Consistency Regularization (STCR). The LCPT automates prompt engineering through data-driven optimization, while the STCR module aligns the category-wise similarity distributions of learnable and hand-crafted prompts under random category subset sampling to alleviate semantic topology drift. Comprehensive experiments and analyses on the OV-AVEBench demonstrate that our SIMPLE framework achieves state-of-the-art performance on the seen and unseen splits, while using roughly half the trainable parameters and roughly one-third the training time.
Open-vocabulary audio-visual event localization (OV-AVEL) grounds a text-queried event in time from video, audio, and language, with a reliability-aware asymmetric distillation framework.
Yi Xu, Cheng Chen, W. Lei· Proceedings of the 34th ACM...· 0 citations
ProCAP is proposed, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine...
H. Abbas, Fatemeh Daneshfar, M. Abdar· arXiv.org· 0 citations
Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) requires models to localize audio-visual events while predicting explicit categories for events unseen during training. Current baselines adopt a text-centered pairwise modality decision strategy. However, it lacks unified modeling of multimodal semantic consist...
Bo-Han Su, Sheng-Lan Liu, Chi Liu et al.· Proceedings of the 34th ACM...· 0 citations
Prompt learning adapts vision-language models such as CLIP by optimizing a short learnable prompt. The cross-entropy objective pulls this prompt toward base-class specialization, whereas generalization to unseen classes benefits from staying close to CLIP's zero-shot feature space. Because both objectives act on the sa...
Audio-visual generalized zero-shot learning leverages the synergy between auditory and visual modalities to recognize instances from seen and unseen categories. However, existing methods usually rely on a rigid alignment paradigm, mapping complex multi-modal features to singular class category. This leads to semantic i...
Bao-Yuan Gao, Shi-Guang Liu· Proceedings of the 34th ACM...· 0 citations
P2CLIP (Prompt-guided CLIP), which aims to bridge the gap by exploring the impact of prompts across three aspects: category, video, and frame, and proposes a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a b...