ORAV Bench for Omni Reference Audio-Video Generation is introduced, comprising 380 task instances with 2-10 references, 9 semantic roles, and 30 role compositions, and a reference-aware pairwise protocol that prepares visual and auditory evidence, compares the intended contribution of each reference, and checks the overall verdict in both presentation orders.
Abstract
Audio-video generation using heterogeneous multimodal references has emerged as a new challenge, requiring both compositional control over generation and grounded understanding of multimodal context. In this paper, we introduce ORAV Bench for Omni Reference Audio-Video Generation, comprising 380 task instances with 2-10 references, 9 semantic roles, and 30 role compositions. Instructions specify the relationships among references; the media supply the identities, dynamics, and audio characteristics to be realized. To evaluate these open-ended outputs, we develop a reference-aware pairwise protocol that prepares visual and auditory evidence, compares the intended contribution of each reference, and checks the overall verdict in both presentation orders. On held-out instances, it achieves 86.08% effective agreement with human judgments. Across 5 frontier systems, overall rankings conceal distinct strengths across reference compositions. A recurring failure is to reproduce unintended source content in place of the requested result, despite closely resembling a reference. Reproducible pointwise diagnostics of quality, reference affinity, and speech reveal distinct dimensions of model behavior. ORAV thus offers a benchmark for tracking progress toward controllable, compositional, and reference-faithful audio-video generation.
This work introduces PEBench, the first unified benchmark for direct PE evaluation across text-to-video, image-to-video, and reference-to-video prompt enhancement, and develops PEBench evaluation, an evidence-grounded framework that combines modality-aware fact extraction with rubric-based assessment across 24 criteria...
Ya-Wen Shao, Yu-Bo Zhu, Zi-Yun Dai et al.· 1 citation
MSR (Multiple Subject Reference), a slot-aware conditioning scheme for LTX-based video generation, is presented and development observations suggest reduced reference confusion relative to an earlier continuous-reference baseline.
VideoVista is presented, a versatile multilingual and multidomain video comprehension benchmark, built via scalable, quality-controlled automatic QA generation and manual refinement, and released at https://github.com/HITsz-TMG/VideoVista.
Yunxin Li, Xin-Yu Chen, Zhenran Xu et al.· IEEE Transactions on Pattern...· 0 citations
A comprehensive study spanning data, benchmarking, and reward modeling, which develops SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips and achieves the strongest agreement with human ratings among the evaluated methods.
Kai-Si Guan, Ying Ba, Wen-Jing Wang et al.· 0 citations
VIBE is introduced, a novel text-and-video-to-music (T+V2M) generation model that leverages a depth-wise cross-layer conditioning mechanism that dynamically bridges the planning and diffusion refinement heads and a comprehensive reward modeling taxonomy, optimizing for both hard, verifiable constraints and soft, subjec...
Generative video systems can produce short clips from textual and visual instructions, yet their ability to preserve the content of a human reference remains uncertain. This exploratory study examines 160 videos generated with Doubao-Seedance-2.0 from 40 human-made short-form videos drawn from Douyin and Xiaohongshu. T...
Chloe Zihan Jin· Communications in Humanities...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.