Skip to content
Preprint

SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization

Oct 2026 · 0 citations · 75 references
Computer Science

TL;DR

This paper proposes SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams and achieves state-of-the-art performance on the seen and unseen splits.

Abstract

Open-vocabulary audio-visual event localization aims to infer explicit segment-level categories for both seen and unseen audio-visual events. Existing solutions rely heavily on hand-crafted prompt templates, whose predictions are highly sensitive to subtle wording variations. Learnable prompts remove this manual tuning burden, but when trained solely on seen events, they tend to overfit to seen categories and distort the inter-event semantic topology required for generalization to unseen categories, a failure we term semantic topology drift. In this paper, we propose SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams. Specifically, our SIMPLE combines Learnable Continuous Prompt Tuning (LCPT) with Semantic Topology Consistency Regularization (STCR). The LCPT automates prompt engineering through data-driven optimization, while the STCR module aligns the category-wise similarity distributions of learnable and hand-crafted prompts under random category subset sampling to alleviate semantic topology drift. Comprehensive experiments and analyses on the OV-AVEBench demonstrate that our SIMPLE framework achieves state-of-the-art performance on the seen and unseen splits, while using roughly half the trainable parameters and roughly one-third the training time.

View source

Similar papers

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

ProCAP is proposed, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine...

H. Abbas, Fatemeh Daneshfar, M. Abdar · 0 citations
Book Open access Nov 2026

Spherical Consistency and Test-time Adaptation for Open-Vocabulary Audio-Visual Event Localization

Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) requires models to localize audio-visual events while predicting explicit categories for events unseen during training. Current baselines adopt a text-centered pairwise modality decision strategy. However, it lacks unified modeling of multimodal semantic consist...

Bo-Han Su, Sheng-Lan Liu, Chi Liu et al. · 0 citations
Preprint Nov 2026

Decoupled Mode Connectivity for Base-to-Novel Generalization in Vision-Language Models

Prompt learning adapts vision-language models such as CLIP by optimizing a short learnable prompt. The cross-entropy objective pulls this prompt toward base-class specialization, whereas generalization to unseen classes benefits from staying close to CLIP's zero-shot feature space. Because both objectives act on the sa...

Imad Eddine Marouf, Khalid Oublal, Enzo Tartaglione et al. · 0 citations
Book Open access Nov 2026

Sight, Sound, and Sense: Unfolding Textual Clouds for Audio-Visual Generalized Zero-Shot Learning

Audio-visual generalized zero-shot learning leverages the synergy between auditory and visual modalities to recognize instances from seen and unseen categories. However, existing methods usually rely on a rigid alignment paradigm, mapping complex multi-modal features to singular class category. This leads to semantic i...

Bao-Yuan Gao, Shi-Guang Liu · 0 citations
Preprint Oct 2026

Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition

P2CLIP (Prompt-guided CLIP), which aims to bridge the gap by exploring the impact of prompts across three aspects: category, video, and frame, and proposes a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a b...

Hai-Chen He, Wei-Bin Liu, Wei-Wei Xing · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.