RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer, mitigates scalar drift, and provides a robust and interpretable reward signal for RL in video generation.
Abstract
Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.
RubricRM is proposed, a pairwise generative reward modeling framework that first produces an input-specific rubric with evaluation dimensions, weights, and scoring criteria, and then applies the rubric to score candidate images.
Zi-Jian Kan, Wei Wang, Long Luo et al.· 0 citations
The Thinking Reward Model (TRM), a paradigm that explicitly determines what matters for each case before judging how well the candidate performs, is proposed, which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards.
Xue-Yuan Bai, Zhen-Chen Tang, Yang Shi et al.· 0 citations
Hindsight Reward Editing (HiRE) bridges the broad knowledge of foundation representation models with physical control awareness, by contrasting successful and failed trajectories in hindsight, and consistently outperforms other reward recipes by delivering dense, control-aware feedback that prevents value function coll...
Haoyi Niu, Zhen Han, Yu-Feng Ji et al.· arXiv.org· 0 citations
This work proposes Step-Aware Annealing (SAA), a plug-and-play reward sharpening mechanism that progressively increases reward curvature during training, amplifying subtle quality differences among high-scoring samples while preserving stability in early learning.
This work proposes LaRes, a novel hybrid framework that achieves efficient policy learning through reward function search by leveraging large language models to generate the reward function population, guiding RL in policy learning.
Pengyi Li, Hongyao Tang, Jinbin Qiao et al.· Neural Information Processin...· 7 citations
This work proposes TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE), and constructs a unified four-paradigm dataset with trajectory progress scoring, video-QA answer quality scoring, and their pairwise counterparts.
Yi-Dong Wang, Yan Zhan, Ziteng Feng et al.· 0 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.