Skip to content

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

Sep 2026 · 1 citation · 31 references
Computer Science

TL;DR

RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer, mitigates scalar drift, and provides a robust and interpretable reward signal for RL in video generation.

Abstract

Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.

View source

Similar papers

Preprint Aug 2026

RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing

RubricRM is proposed, a pairwise generative reward modeling framework that first produces an input-specific rubric with evaluation dimensions, weights, and scoring criteria, and then applies the rubric to score candidate images.

Zi-Jian Kan, Wei Wang, Long Luo et al. · 0 citations
Preprint Sep 2026

Think Before You Score: Thinking Reward Model for Visual Generation

The Thinking Reward Model (TRM), a paradigm that explicitly determines what matters for each case before judging how well the candidate performs, is proposed, which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards.

Xue-Yuan Bai, Zhen-Chen Tang, Yang Shi et al. · 0 citations
Sep 2026

HiRE: Hindsight Reward Editing for Policy Finetuning

Hindsight Reward Editing (HiRE) bridges the broad knowledge of foundation representation models with physical control awareness, by contrasting successful and failed trajectories in hindsight, and consistently outperforms other reward recipes by delivering dense, control-aware feedback that prevents value function coll...

Haoyi Niu, Zhen Han, Yu-Feng Ji et al. · 0 citations

DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards

This work proposes Step-Aware Annealing (SAA), a plug-and-play reward sharpening mechanism that progressively increases reward curvature during training, amplifying subtle quality differences among high-scoring samples while preserving stability in early learning.

Unknown authors · 0 citations
2025

LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search

This work proposes LaRes, a novel hybrid framework that achieves efficient policy learning through reward function search by leveraging large language models to generate the reward function population, guiding RL in policy learning.

Pengyi Li, Hongyao Tang, Jinbin Qiao et al. · 7 citations
Preprint Aug 2026

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models

This work proposes TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE), and constructs a unified four-paradigm dataset with trajectory progress scoring, video-QA answer quality scoring, and their pairwise counterparts.

Yi-Dong Wang, Yan Zhan, Ziteng Feng et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.