Step-wise Rubric Rewards for LLM Reasoning

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves language-model reasoning using final-answer correctness, but lacks supervision for intermediate steps. Rubric-based methods such as Rubrics as Rewards (RaR) provide finer-grained criteria, yet collapse rubric scores into a response-level scalar. This loses criterion-level structure, rewards flawed steps in correct responses, penalizes valid steps in incorrect responses, and can encourage unbounded self-correction. In 1,000 sampled problems, 18.2% of steps in answer-correct responses are wrong, while 49.9% of steps in answer-incorrect responses are correct. We propose Step-wise Rubrics as Rewards (SRaR), which (i) uses an LLM judge to assign rubric items to reasoning steps, (ii) normalizes step scores across rollouts to focus learning on quality differences, and (iii) combines step-wise and outcome rewards through a decoupled advantage estimator that preserves a stable outcome baseline. We also construct a 16K-problem rubric dataset by contrastively distilling criteria from verified correct and flawed reasoning paths. Across six math benchmarks, SRaR improves average accuracy over RaR by 3.57 and 2.75 points on Qwen3-8B-Non-Thinking and Qwen3-32B-Non-Thinking, respectively. On AIME 2025, it increases the Faithful Reasoning Rate from 34.5% to 46.7% and reduces self-correction looping from 48.1% to 26.5%. SRaR also achieves the best results on three of four code-generation benchmarks.