Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While \emph{value evolution along trajectory} characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, w...
Fu-Ting Wang, Chen-Xi Li, Jian-Hao Yan et al.· 0 citations
This work compares the student view and training fixed within each scale against three abstractions compiled offline, a named strategy, a method-independent framing, and a problem category, and against an answer-only control that keeps the destination but removes the path.
Kang-Hui Tian, Si-Yuan Liu, Tianxiang Jiang et al.· 2 citations
Value Flattening is identified as an important yet overlooked failure mode of critic learning in standard PPO and a simple sparse supervision strategy can mitigate it; SParse Proximal Policy Optimization is introduced, which applies the value loss to only a few well-separated states in each response to mitigate both ef...
Yi-Zhuo Li, Jian-Hao Yan, Yun Luo et al.· 1 citation
This work introduces a student reference KL loss and mask the advantages of special termination tokens to mitigate the problem of excessive generation length and frequent truncation, and improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the...
Hao-Nan He, Hao Lei, Yun Luo et al.· 5 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.