Omni Demand Understanding is established as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from multimodal and conversational context, and ODU is evaluated along five dimensions, covering both single-turn and multi-turn interactions.
Abstract
Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather than carefully composed text prompts. However, existing benchmarks of interactive capabilities still focus primarily on response quality, leaving a more fundamental question underexplored: can a model correctly infer the user's underlying demand from complex multimodal interaction? Real-world user demands are often underspecified in speech and must be inferred from multimodal cues and dialogue history. This inference is further complicated by ambiguous or disfluent expression and noisy acoustic environments. Conversely, request-like speech may not constitute a demand to the assistant, leading to false triggers. We establish Omni Demand Understanding (ODU) as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from multimodal and conversational context. ODU evaluates this capability along five dimensions, covering both single-turn and multi-turn interactions. We construct ODU-Bench using a challenge-driven taxonomy, taxonomy-guided agentic video generation, and human-recorded interactions, followed by media-grounded annotation and human verification. We evaluate 14 native MLLMs. Even the strongest, Gemini 3.1 Pro, recovers only 44.7% of key information that must be inferred from visual, acoustic, or conversational context. Moreover, 11 of the 14 models exhibit false-trigger rates above 50% on non-demand scenarios. These results reveal a systematic capability gap in current MLLMs'ability to infer contextual user demands. We hope ODU can establish the evaluation of a previously underexplored yet essential capability in multimodal interaction: correctly understanding user demands before generating an appropriate response.
OmniSmartHome is introduced, a multimodal smart-home benchmark where each spoken request is paired with the surrounding visual and spatial-audio context, providing complementary cues to disambiguate underspecified requests.
Jihoo Jung, Suho Yoo, Jeongsoo Choi et al.· 0 citations
Findings show that historical evidence use should be evaluated under state variation, rather than inferred from outcome-only correctness, and that open-weight models rely heavily on context continuity and are reluctant to spontaneously switch to tool-mediated retrieval when provenance fails.
Yu Liu, Wen-Xiao Zhang, Cheng Hu et al.· 0 citations
It is revealed that substantial priors can be learned solely from the text modality and that using a textual caption instead of the video yields performance on par with Vanilla SFT, revealing the limitations of current MLLMs when it comes to social understanding.
Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera et al.· 0 citations
Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform strongly but rely on task-specific training. Recent multimodal large language models (MLLMs) offer a promising route to training-free HOI detection through their broad vi...
Zhao-Lin Cai, Hui-Yu Duan, Liu Yang et al.· 0 citations
S$^3$-Bench is introduced, a systematic evaluation framework covering 10 major disciplines, consisting of a Knowledge set for speech question-answering and a Dialogue set for multi-turn progressive interactions with simulated user agents that systematically characterize the common challenges and performance tradeoffs o...
He-Yang Liu, Jia-Yi Huang, Wen Xiao et al.· 0 citations
Hear2Act is introduced, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes that show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do...
Xin-Yi Liu, H. Nayyeri, Dilek Hakkani-Tur et al.· 3 citations· ⚡1
The new ChartNet training dataset could improve the accuracy of vision-language models that help analyze business trends or interpret scientific figures.