Skip to content
Book Open access

Sight, Sound, and Sense: Unfolding Textual Clouds for Audio-Visual Generalized Zero-Shot Learning

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 44 references

Abstract

Audio-visual generalized zero-shot learning leverages the synergy between auditory and visual modalities to recognize instances from seen and unseen categories. However, existing methods usually rely on a rigid alignment paradigm, mapping complex multi-modal features to singular class category. This leads to semantic information loss and misalignment. To break this bottleneck, we propose a novel Textual Cloud strategy. It elaborates singular category into a multi-faceted semantic landscape, including visual, auditory, and contextual cues. These facets represent physical appearance, acoustic signatures, and abstract definitions, respectively, forming a set of orthogonal and distinct semantic subspaces. Furthermore, to orchestrate these disentangled semantic cues, we introduce a Gated Global Attention (GGA) module. GGA dynamically recalibrates the importance of each aspect, preventing over-reliance on any single facet. It simultaneously allows each textual cloud aspect to be enriched by the others, fusing comprehensive semantics to guide the zero-shot recognition. Extensive experiments on VGGSound, UCF-101, and ActivityNet benchmarks demonstrate that our method establishes new state-of-the-art performance, achieving improvements in Harmonic Mean metric by margins of 4.17% and 5.22% on VGGSound and ActivityNet, respectively. Our source code and supplementary material are available at: https://github.com/Asia-Leee/Textual-Cloud.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.