Rewarding Better Thinking for LLM Preference Alignment

Xu-Bo Liu Wenya Guo Ruxue Yan Xinying Qian Ying Zhang

Jul 2026 · arXiv.org · Vol abs/2607.19824 · 0 citations

Computer Science

TL;DR

Think Checklist Reward (TCR), a process-oriented reward for RL-based preference alignment that converts preference pairs into sample-specific thinking checklists and uses them to evaluate whether the generated reasoning trace addresses the preference-implied considerations.

Abstract

LLM preference alignment aims to optimize models toward human preferences across diverse user instructions. Reinforcement learning has become a major post-training approach for this goal, but existing proxy rewards are often outcome-level, mainly evaluating the final response while providing limited guidance for the reasoning trajectory. This can make credit assignment coarse when multiple responses receive similar final scores, leaving trajectory-level preferences under-specified. To address this limitation, we propose Thinking Checklist Reward (TCR), a process-oriented reward for RL-based preference alignment. TCR converts preference pairs into sample-specific thinking checklists and uses them to evaluate whether the generated reasoning trace addresses the preference-implied considerations. To reduce overlap with outcome-level supervision, TCR further introduces an exponential moving average (EMA) residual formulation to isolate a complementary thinking surplus beyond what is predictable from the outcome reward. Experiments on five models from three model families show that TCR consistently improves alignment performance across diverse benchmarks, with ablations further validating the importance of EMA-based residual formulation and sample-specific checklist supervision.

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

Procedural Fairness Failures in RLHF from Preference Averaging

The results show that procedural fairness failures in alignment can arise from structural design choices in reward learning, even in controlled, noise-free settings, with direct implications for large language models and agentic systems, where biased reward models can compound inequities across sequential decisions.

M. Gopinadh, Karthik Kamuju, K. Avinash et al. · 0 citations

Jul 2026

Measuring Reward-Seeking via Contrastive Belief Updates

The results indicate that RL can increase reward-seeking over the course of training, producing models that may act against their developers'intentions when they believe that doing so leads to higher reward.

Axel Højmark, Jérémy Scheurer, Evgenia Nitishinskaya et al. · arXiv.org · 2 citations

Preprint Aug 2026

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

This work proposes \methodname, a stability-guided active-set controller for controlled objective admission, a stability-guided active-set controller for controlled objective admission in reward-vector RLHF, which positions objective-entry timing as a concrete control variable in reward-vector RLHF.

Yong-Qi Tong, Z. Zhang, Ruirui Wang et al. · 1 citation

Preprint Aug 2026

To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization

This work argues that, given a sufficiently diverse user population, a curriculum naturally emerges between easy- and hard-to-optimize reward models, and proposes CurriPO, which grows a tree-structured curriculum to accommodate diverse user-specific objectives, covering the population in a single traversal.

Taehyung Kim, Jongeun Choi · 0 citations

#machine learning Preprint Sep 2026

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock ov...

Le-Qi Zheng, Jin-Bo Su, Fang Niu et al. · 2 citations

Preprint Aug 2026

Instruction Quality Matters: Refining Instructions for Effective Preference Learning

This work introduces an instruction-refinement pipeline that selects weak instructions using reward signals and revises them with rubric-guided LLM feedback, improving preference data without discarding examples, and indicates that instruction refinement raises achievable response quality and complements response-centr...

Seohyeong Lee, Hwaran Lee, Buru Chang · 0 citations