Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving"who says what"is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVL...
Jihoo Jung, Youngjoon Jang, Joon Son Chung· 1 citation
This work aims to improve how audio-visual large language models (AVLLMs) associate speech with the correct visible speaker in multi-speaker scenes. We find that current AVLLMs frequently fail at this task, and analyze the nature of these failures. To this end, we construct a synthetic diagnostic dataset in which multi...
Jihoo Jung, Youngjoon Jang, Hyebin Cho et al.· arXiv.org· 0 citations
A prototype-structured sign embedding space is learned from continuous video annotated with signs, where each learnable prototype corresponds to a sign class, enabling the matching between dictionary exemplars and continuous sign instances.
Ryan Wong, Youngjoon Jang, Liliane Momeni et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.