Skip to content
Preprint

VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

Aug 2026 · 6 citations · ⚡ 1 influential · 67 references
Computer Science

TL;DR

VA-Judger is proposed, a chain-of-thought omni-reward model that learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals.

Abstract

Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.

View source

Similar papers

Preprint Aug 2026

Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation

Video generation is central to AI-powered content creation. Aligning generated videos with human preferences is a key criterion for evaluating generation quality. Despite significant progress in visual quality, three key challenges remain. First, the reliability of reward signals is constrained by the quality of human...

Nai-Xin Zhai, Weihua Cheng, De-Xu Yu et al. · 0 citations
Preprint Oct 2026

TiTok: Audio-Visual LLM for Multi-Segment Temporal Grounding

TiTok is presented, an audio-visual large language model (AV-LLM) that localizes an arbitrary number of temporal event segments for each query, and the Time Token Interleaving (TTI) method is introduced, which explicitly injects special time tokens into the audio-visual stream to align input-side temporal perception wi...

Eunji Shin, Dahyun Choi, Seungyeon Jo et al. · 0 citations
Preprint Oct 2026

Token-Level Video Reinforcement Learning

Reinforcement learning (RL) for video generation usually assigns one scalar reward to an entire sampled video. Yet a video is not uniformly flawed: some visual tokens may already satisfy the prompt, whereas others require correction. A scalar reward cannot localize errors, causing optimization to perturb satisfactory t...

Yi-Fan Wang, G. Qian, Yan-Yu Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer, mitigates scalar drift, and provides a robust and interpretable reward signal for RL in video generation.

Zhen-Chen Tang, Yang Li, Songlin Yang et al. · 1 citation
Preprint Sep 2026

Think Before You Score: Thinking Reward Model for Visual Generation

The Thinking Reward Model (TRM), a paradigm that explicitly determines what matters for each case before judging how well the candidate performs, is proposed, which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards.

Xue-Yuan Bai, Zhen-Chen Tang, Yang Shi et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.