Preprint

RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

Chenglong Wang Ziming Zhu Yifu Huo Bei Li Qiao-Zhi He Yan Ding Xiaoyang Hao Yuxin Gao Tian-Hua Zhou Xiao-Wen Chang Tongran Liu Jing-Bo Zhu

Aug 2026 · 2 citations

Computer Science

TL;DR

RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses.

Abstract

Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.

arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

RMB: Reward Model Boosting Mitigates Reward Hacking

Reinforcement Learning from Human Feedback (RLHF) is a powerful technique for aligning large language models (LLMs) with human preference. However, it often suffers from the reward hacking issue, where policy optimization improves the proxy reward model while actually degrading performance with respect to the true huma...

Jia-Bin Fan, De-Zhi Ye, Yong-Chang Hao et al. · 0 citations

#machine learning Preprint Sep 2026

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock ov...

Le-Qi Zheng, Jin-Bo Su, Fang Niu et al. · 2 citations

Conference Open access Sep 2026

Adaptive Reward Design in Reinforcement Learning: A Taxonomy and Survey

A unified view of ARD in RL is provided by introducing a taxonomy, organized by the primary driver of the reward variation, that distinguishes external-feedback-driven reward updates from reward adaptations driven by endogenous within-run signals and those conditioned on exogenous context signals.

Raphaela Baybas, Carlo D'Eramo, Philipp Brune · Proceedings of the Thirty-Fi... · 0 citations

#artificial intelligence Preprint Sep 2026

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer, mitigates scalar drift, and provides a robust and interpretable reward signal for RL in video generation.

Zhen-Chen Tang, Yang Li, Songlin Yang et al. · 1 citation

#natural language process... Preprint Aug 2026

Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning

Adaptive Multi-Reward Projection is proposed, a lightweight online method that reallocates aggregation weights using three signals, relative shortfall, reward volatility, and recent progress, increasing pressure on lagging, unstable, or stagnant dimensions while relieving saturated ones.

Yu Yuan, Yao Fan, Lili Zhao et al. · 2 citations

Preprint Sep 2026

HiRE: Hindsight Reward Editing for Policy Finetuning

Hindsight Reward Editing (HiRE) bridges the broad knowledge of foundation representation models with physical control awareness, by contrasting successful and failed trajectories in hindsight, and consistently outperforms other reward recipes by delivering dense, control-aware feedback that prevents value function coll...

Haoyi Niu, Zhen Han, Yu-Feng Ji et al. · 0 citations