Verifiable rewards improve language models through reliable task-level feedback, but methods based on Group Relative Policy Optimization (GRPO) apply a sequence-level advantage uniformly across all tokens. This coarse credit assignment reinforces or penalizes entire responses without identifying which local decisions to preserve, reinforce, or revise. Conversely, evidence-conditioned self-distillation provides denser token-level supervision, yet teacher imitation can transfer stylistic artifacts and miscalibrated confidence that destabilize training when misaligned with task success. We introduce VERPO, which converts evidence-conditioned guidance into reward-aligned token-level credit assignment while retaining the outcome objective. VERPO decomposes teacher guidance into an evidence-free reference term and signed, evidence-induced corrections at each token. A stopped controller combines selective acceptance, token-wise localization, and cost-aware scaling by balancing alignment with the local GRPO update direction against Fisher movement cost. Furthermore, we introduce Fisher Evidence Contrast (FEC), which attenuates nuisance shifts along an estimated evidence-presence direction through a regularized projection. Across five scientific reasoning and tool-use tasks, VERPO prevents optimization collapse and consistently achieves the highest multi-task average across model backbones, yielding marked improvements particularly on smaller models over strong baselines. Qualitative diagnostics confirm that token acceptance selectively targets reasoning bottlenecks consistent with local reward alignment and Fisher movement cost.
Reward-Aligned Reweighting for On-Policy Distillation (R$^{2}$-OPD), which uses outcome agreement and the magnitude of teacher--student disagreement to continuously reallocate teacher supervision, achieves the highest average accuracy among the compared training methods in both cross-size and same-size distillation.
Hao Xu, Junwei Su, Lan-Song Diao et al.· arXiv.org· 0 citations
UniOPSD (Unified On-Policy Self-Distillation) is introduced, which unifies these feedback sources through adaptive local credit arbitration and constructs comparable credit estimates from environmental returns and successful-peer hindsight at shared interaction anchors.
Zeng-Huang Fu, Zhao-Yang Li, Qiu-Yuan Ai et al.· arXiv.org· 0 citations
Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy, is developed.
Jia-Yan Fu, Hang Xu, Yong Zhang et al.· arXiv.org· 0 citations
Group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance, and demonstrates that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance.
Zhu Zhang, Ji-Xun Wang, Xiao-An Xu et al.· 3 citations
A unified temporal-credit view of token-level OPD and sequence-level OPD is established, showing that practical token-level OPD can be interpreted as a temporal approximation to the sequence-level reverse-KL gradient.
Shi-Qi Liu, Ze-Yu He, Le-Tian Tao et al.· 3 citations
Requiring only outcome rewards and no process-level annotation, CBPO provides a practical solution for fine-grained credit assignment in tool-integrated agent training and consistently outperforms state-of-the-art policy-optimization and branch-based methods.
Ying Wang, Changlin Qiu, Bang Lin et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.