Skip to content

CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models

Sep 2026 · 0 citations
Computer Science

TL;DR

CoRe is introduced, a co-evolving reward framework that treats latent-space alignment as a dynamic interaction between the generator and the reward model, and consistently improves generation quality over both the pretrained model and prior alignment methods.

Abstract

Latent reward models (LRMs) enable efficient alignment of video diffusion models by scoring intermediate states directly in latent space. However, we find that optimizing against a fixed latent reward rapidly leads to latent reward hacking: the predicted reward stays high while perceptual and motion quality deteriorate. Our analysis identifies distributional escape as the central cause: within a few hundred updates, the generator moves beyond the reward model's training support, where its scores no longer reflect video quality. Based on this insight, we introduce CoRe, a co-evolving reward framework that treats latent-space alignment as a dynamic interaction between the generator and the reward model. Rather than optimizing against a stationary proxy, CoRe continually refits the reward model on the generator's current samples while anchoring it to real-video preferences, so the generator cannot gain reward by drifting away from the data. On Wan2.1-T2V-1.3B, experiments show that CoRe consistently improves generation quality over both the pretrained model and prior alignment methods, while avoiding the quality collapse of fixed-reward optimization.

View source

Similar papers

Diffusion Reward Models

Downstream RLHF experiments additionally show that using DRM as the training-time reward leads to improved policy performance, directly validating the practical benefit of diffusion-based reward modeling for RLHF training.

Xiang-Yang Wang, Bing-Xiang He, Ze-Yuan Liu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer, mitigates scalar drift, and provides a robust and interpretable reward signal for RL in video generation.

Zhen-Chen Tang, Yang Li, Songlin Yang et al. · 1 citation
Preprint Aug 2026

On-Policy Self-Distillation in Diffusion Models

The results support on-policy self-distillation as an efficient and analyzable approach to diffusion post-training by converting image-level reward guidance into explicit and continually refreshed intermediate supervision, thereby opening a path toward more efficient and diagnosable alignment.

Weina Zhou, Xiongwei Zhu, Ling-Dong Kong et al. · 9 citations
Preprint Aug 2026

VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

Variance-reduced Guidance and Adaptive Selection (VGAS), a simple yet effective inference-time framework that reduces the variance of the guidance estimate for both reward types, applies the reward tilting in the clean-token logits, where the pretrained schedule is preserved, and sets the selection temperature per step...

Kwanyoung Kim · 1 citation
#natural language process... Preprint Aug 2026

Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning

Adaptive Multi-Reward Projection is proposed, a lightweight online method that reallocates aggregation weights using three signals, relative shortfall, reward volatility, and recent progress, increasing pressure on lagging, unstable, or stagnant dimensions while relieving saturated ones.

Yu Yuan, Yao Fan, Lili Zhao et al. · 3 citations
#machine learning Preprint Oct 2026

DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA)...

Zheng-Lin Wan, Jia-Yu Chen, Zhengfeiyang Zhang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.