Skip to content

ORAV: Benchmarking Audio-Video Generation from Multimodal Contexts

Sep 2026 · arXiv.org · Vol abs/2609.34843 · 0 citations · 48 references
Computer Science

TL;DR

ORAV Bench for Omni Reference Audio-Video Generation is introduced, comprising 380 task instances with 2-10 references, 9 semantic roles, and 30 role compositions, and a reference-aware pairwise protocol that prepares visual and auditory evidence, compares the intended contribution of each reference, and checks the overall verdict in both presentation orders.

Abstract

Audio-video generation using heterogeneous multimodal references has emerged as a new challenge, requiring both compositional control over generation and grounded understanding of multimodal context. In this paper, we introduce ORAV Bench for Omni Reference Audio-Video Generation, comprising 380 task instances with 2-10 references, 9 semantic roles, and 30 role compositions. Instructions specify the relationships among references; the media supply the identities, dynamics, and audio characteristics to be realized. To evaluate these open-ended outputs, we develop a reference-aware pairwise protocol that prepares visual and auditory evidence, compares the intended contribution of each reference, and checks the overall verdict in both presentation orders. On held-out instances, it achieves 86.08% effective agreement with human judgments. Across 5 frontier systems, overall rankings conceal distinct strengths across reference compositions. A recurring failure is to reproduce unintended source content in place of the requested result, despite closely resembling a reference. Reproducible pointwise diagnostics of quality, reference affinity, and speech reveal distinct dimensions of model behavior. ORAV thus offers a benchmark for tracking progress toward controllable, compositional, and reference-faithful audio-video generation.

View source

Similar papers

Preprint Oct 2026

Towards Unified Evaluation of Prompt Enhancers for Video Generation

This work introduces PEBench, the first unified benchmark for direct PE evaluation across text-to-video, image-to-video, and reference-to-video prompt enhancement, and develops PEBench evaluation, an evidence-grounded framework that combines modality-aware fact extraction with rubric-based assessment across 24 criteria...

Ya-Wen Shao, Yu-Bo Zhu, Zi-Yun Dai et al. · 1 citation
Sep 2026

MSR: Multiple Subject Reference for Video Generation

MSR (Multiple Subject Reference), a slot-aware conditioning scheme for LTX-based video generation, is presented and development observations suggest reduced reference confusion relative to an earlier continuous-reference baseline.

Guan-Nan Li, Jia-Ji Chen, Jing-Yuan Liao et al. · 0 citations
#small language model Open access Oct 2026

VideoVista: Benchmarking Diverse and Complex Video-Language Interaction for MLLMs.

VideoVista is presented, a versatile multilingual and multidomain video comprehension benchmark, built via scalable, quality-controlled automatic QA generation and manual refinement, and released at https://github.com/HITsz-TMG/VideoVista.

Yunxin Li, Xin-Yu Chen, Zhenran Xu et al. · 0 citations
Preprint Oct 2026

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

A comprehensive study spanning data, benchmarking, and reward modeling, which develops SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips and achieves the strongest agreement with human ratings among the evaluated methods.

Kai-Si Guan, Ying Ba, Wen-Jing Wang et al. · 0 citations
#artificial intelligence Preprint Aug 2026

VIBE: Video Instruction-aligned Background music gEneration

VIBE is introduced, a novel text-and-video-to-music (T+V2M) generation model that leverages a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints and soft, subjec...

Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj et al. · 0 citations
#generative ai Review Sep 2026

Reference Fidelity in AI-Generated Short-Form Video Across Four Prompt Packages

Generative video systems can produce short clips from textual and visual instructions, yet their ability to preserve the content of a human reference remains uncertain. This exploratory study examines 160 videos generated with Doubao-Seedance-2.0 from 40 human-made short-form videos drawn from Douyin and Xiaohongshu. T...

Chloe Zihan Jin · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.