RMEval: Bridging Reward Model Evaluation and Policy Evaluation through Ranking Consistency

Abstract

Reward Models (RMs) are widely used to guide the alignment of Large Language Models (LLMs) in Reinforcement Learning from Human Feedback (RLHF), and their quality has a substantial impact on the performance of trained policy models. In current practice, reward models are predominantly evaluated using local metrics such as pairwise accuracy on validation sets. However, recent studies have shown that the accuracy metric exhibits only weak correlation with downstream policy performance. To bridge this gap, we propose RMEval, a benchmark that revisits reward model evaluation through the lens of global ranking consistency. Specifically, we conduct the standard RLHF training to generate a sequence of policy checkpoints with progressively improving performance. We then construct our evaluation sets by sampling responses from these checkpoints, establishing a ground-truth ranking based on downstream policy evaluation. By scoring these responses with a candidate reward model, we measure the consistency between the ranking induced by the reward model and the ground-truth ranking defined by policy performance, using the consistency as our evaluative metric. Through extensive empirical analysis, we evaluate existing reward models on RMEval and demonstrate that our metric exhibits a strong correlation with downstream policy performance, with Pearson coefficients exceeding 0.6 across multiple domains.