This work introduces Trajectory-to-Step Policy Optimization (T2SPO), a method that uses past interaction trajectories to provide step-level feedback for policy learning and shows that T2SPO consistently improves overall task success over GRPO.
Bo-Wen Zhang, Jun-Wei He, Mao-Qi Liu et al.· 1 citation
Rubric-based reinforcement learning (Rubric-RL) trains language models where no verifier exists. A judge checks each criterion of a rubric, and the verdicts are aggregated into a reward, most often by a weighted sum. We show that this additive aggregation is the weak point. Under a sum, criteria compensate for one anot...
Mao-Qi Liu, Jun-Wei He, Bo-Wen Zhang et al.· 0 citations
CoRT is proposed, a token-level credit weighting method for rubric-conditioned GRPO that uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt, and suggests that policy-internal counterfactual likelihood contrasts provide an effec...
Bo-Wen Zhang, Jun-Wei He, Wen Wang et al.· arXiv.org· 5 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.