Aug 2026· 1 citation· ⚡ 1 influential· 36 references
Computer Science
Abstract
Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets. Although scalar MSE is statistically valid for estimating the conditional expected return, sparse binary rewards in reinforcement learning with verifiable rewards (RLVR) make critic optimization and calibration especially consequential: small value errors directly distort the scalar advantages used by PPO. We study whether a classification-based training objective can improve this critic signal. HL-Gauss PPO replaces the scalar MSE head with a categorical predictor over a discretized value support, trained by cross-entropy against smoothed HL-Gauss targets. Its output is decoded to a scalar expectation for standard GAE and PPO; the actor update is therefore unchanged and is not distributional. Across mathematical reasoning, tool-augmented math, and Search-R1, and on both Qwen2.5 and Qwen3 backbones, HL-Gauss PPO consistently improves over strong PPO and DAPO baselines. Controls with one-hot, two-hot, and Bernoulli two-bin critics show that neither a larger output head nor binary classification alone explains the gains. On a common collection of reasoning prefixes, HL-Gauss improves Brier score and calibration error and yields more symmetric, lower-variance advantages. These results position categorical value learning as an effective optimization surrogate for PPO critics in RLVR.
Best Practice Critic Optimization (BPCO) is developed, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation and shows that a carefully designed critic provides a reliable alternative to group-relative advantage estimation.
This work proposes two complementary strategies to improve the performance of value function RL: Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; and TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy.
S. Venkatraman, Matthieu Dinot, Laurence Aitchison· 0 citations
Proximal Policy Optimization (PPO) is widely used in reinforcement learning because it provides stable on-policy learning with relatively simple implementation. However, standard PPO uses a fixed clipping threshold, applying the same trust-region width to all training samples even though the reliability of policy updates may vary across states and actions. This paper studies a family of adaptive-clipping PPO variants that replace the constant clipping parameter with sample-dependent thresholds derived from policy and return statistics. We introduce Dual-Adaptive Z-Score PPO (PPO-DAZ3), which dynamically adjusts the clipping interval using two signals: action confidence, derived from the probability of the selected action under the current policy, and update confidence, measured from normalized advantage magnitude. This formulation enables persample adaptation while preserving the standard PPO training pipeline. Experiments on three classic reinforcement learning benchmarks-LunarLander-v3, CartPole-v1, and Acrobotv1-show that adaptive clipping can improve learning performance in some environments. PPO-DAZ3 achieved the strongest results on LunarLander-v3, while the smooth advantage-based variant PPO-AZ3S performed best on Acrobotv1. On CartPole-v1, standard PPO remained competitive, indicating that adaptive clipping is most beneficial on more challenging control tasks.
Taha M. Mahmoud, N. Kaabouch· 2026 6th International Confe...· 0 citations
MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function across auxiliary tasks before RLHF training, is introduced, providing theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization.
AdaKP is an online selector that re-chooses each problem's KP subset over the course of RL training, an entropy proxy that scores a KP by the reduction in next-token entropy it induces in a single inexpensive forward pass, with a provable bound on its truncation bias.