Skip to content
Preprint

Think Before You Score: Thinking Reward Model for Visual Generation

Sep 2026 · 0 citations · 59 references
Computer Science

TL;DR

The Thinking Reward Model (TRM), a paradigm that explicitly determines what matters for each case before judging how well the candidate performs, is proposed, which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards.

Abstract

Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer, mitigates scalar drift, and provides a robust and interpretable reward signal for RL in video generation.

Zhen-Chen Tang, Yang Li, Songlin Yang et al. · 1 citation
Preprint Aug 2026

RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing

RubricRM is proposed, a pairwise generative reward modeling framework that first produces an input-specific rubric with evaluation dimensions, weights, and scoring criteria, and then applies the rubric to score candidate images.

Zi-Jian Kan, Wei Wang, Long Luo et al. · 0 citations
Preprint Aug 2026

VIG: Visual Information Gain as a Reward Signal for Multimodal Chain-of-Thought Compression

Across six main multimodal reasoning benchmarks and three Qwen3-VL-Thinking model sizes, VIG consistently improves the accuracy--efficiency trade-off, supporting the central claim: efficient multimodal reasoning emerges from raising visual information density.

Wen Luo, Xiao-Han Yi, Xiao-Tao Huang et al. · 0 citations
Preprint Aug 2026

Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

It is shown that Plackett-Luce (PL) reward models can train robotic policies from VLM-generated rankings as effectively as pairwise Bradley-Terry, $K$-wise Bradley-Terry, and RL-VLM-F baselines and demonstrate that listwise VLM preference supervision is a competitive and flexible approach to reward learning for reinfor...

Srivalli Katkuri, Maxwell Kawada, J. Wachs · 0 citations
Review Open access Aug 2026

Reinforcement Learning for Multimodal Foundation Models: A Survey

A coherent map of the rapidly expanding landscape of visual RL is provided to provide researchers and practitioners with a coherent map of the rapidly expanding landscape of visual RL and to highlight promising directions for future inquiry.

Weijia Wu, Chen Gao, Joya Chen et al. · 0 citations
Preprint Sep 2026

GDB-Reward: From Evaluation Metrics to Training Rewards for Graphic Design

Text-to-image models excel at natural image synthesis but struggle with graphic design, where success depends on satisfying precise constraints on typography, layout, color, and visual communication. While prompt optimization offers an attractive alternative to expensive diffusion model fine-tuning, learning prompts fo...

Adrienne Deganutti, Purvanshi Mehta, Simon Hadfield et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.