Skip to content

Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction

Sep 2026 · 1 citation · ⚡ 1 influential · 28 references
Computer Science

TL;DR

Omni Demand Understanding is established as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from multimodal and conversational context, and ODU is evaluated along five dimensions, covering both single-turn and multi-turn interactions.

Abstract

Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather than carefully composed text prompts. However, existing benchmarks of interactive capabilities still focus primarily on response quality, leaving a more fundamental question underexplored: can a model correctly infer the user's underlying demand from complex multimodal interaction? Real-world user demands are often underspecified in speech and must be inferred from multimodal cues and dialogue history. This inference is further complicated by ambiguous or disfluent expression and noisy acoustic environments. Conversely, request-like speech may not constitute a demand to the assistant, leading to false triggers. We establish Omni Demand Understanding (ODU) as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from multimodal and conversational context. ODU evaluates this capability along five dimensions, covering both single-turn and multi-turn interactions. We construct ODU-Bench using a challenge-driven taxonomy, taxonomy-guided agentic video generation, and human-recorded interactions, followed by media-grounded annotation and human verification. We evaluate 14 native MLLMs. Even the strongest, Gemini 3.1 Pro, recovers only 44.7% of key information that must be inferred from visual, acoustic, or conversational context. Moreover, 11 of the 14 models exhibit false-trigger rates above 50% on non-demand scenarios. These results reveal a systematic capability gap in current MLLMs'ability to infer contextual user demands. We hope ODU can establish the evaluation of a previously underexplored yet essential capability in multimodal interaction: correctly understanding user demands before generating an appropriate response.

View source

Similar papers

Preprint Sep 2026

OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents

OmniSmartHome is introduced, a multimodal smart-home benchmark where each spoken request is paired with the surrounding visual and spatial-audio context, providing complementary cues to disambiguate underspecified requests.

Jihoo Jung, Suho Yoo, Jeongsoo Choi et al. · 0 citations
#artificial intelligence Preprint Aug 2026

MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents

Findings show that historical evidence use should be evaluated under state variation, rather than inferred from outcome-only correctness, and that open-weight models rely heavily on context continuity and are reluctant to spontaneously switch to tool-mediated retrieval when provenance fails.

Yu Liu, Wen-Xiao Zhang, Cheng Hu et al. · 0 citations
Preprint Aug 2026

Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?

It is revealed that substantial priors can be learned solely from the text modality and that using a textual caption instead of the video yields performance on par with Vanilla SFT, revealing the limitations of current MLLMs when it comes to social understanding.

Koen P. de Vries, Xavier Alameda-Pineda, Estefanía Talavera et al. · 0 citations
Preprint Oct 2026

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Traditional supervised methods perform strongly but rely on task-specific training. Recent multimodal large language models (MLLMs) offer a promising route to training-free HOI detection through their broad vi...

Zhao-Lin Cai, Hui-Yu Duan, Liu Yang et al. · 0 citations
#natural language process... Preprint Sep 2026

$S^3$-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants

S$^3$-Bench is introduced, a systematic evaluation framework covering 10 major disciplines, consisting of a Knowledge set for speech question-answering and a Dialogue set for multi-turn progressive interactions with simulated user agents that systematically characterize the common challenges and performance tradeoffs o...

He-Yang Liu, Jia-Yi Huang, Wen Xiao et al. · 0 citations
Preprint Aug 2026

Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

Hear2Act is introduced, a unified evaluation protocol for text and spoken assistants with 480 persona-grounded scenarios, hidden user concerns, and objectively verifiable outcomes that show that prosody matters when lexical evidence is insufficient, and that audio-capable LLMs can recover information from speech but do...

Xin-Yi Liu, H. Nayyeri, Dilek Hakkani-Tur et al. · 3 citations · ⚡1

Related blog posts

MIT News · Artificial Intelligence Jun 3, 2026

MIT researchers teach AI models to interpret charts

The new ChartNet training dataset could improve the accuracy of vision-language models that help analyze business trends or interpret scientific figures.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.