The results support compact decision models as a practical component of larger agent systems as well as qualitative game, visual interface, and browser demonstrations.
Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residua...
Ze-Feng He, Yun-Kai Chai, Xiao-Yang Chen et al.· 0 citations
Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While \emph{value evolution along trajectory} characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, w...
Fu-Ting Wang, Chen-Xi Li, Jian-Hao Yan et al.· 0 citations
On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through...
Draft-OPD, which uses target-assisted rollout for stable continuations and replays drafting from the verification-exposed error positions, allowing the drafter to learn from target feedback on both accepted and rejected proposals, focusing on the draft-induced errors that limit speculative acceptance.
Value Flattening is identified as an important yet overlooked failure mode of critic learning in standard PPO and a simple sparse supervision strategy can mitigate it; SParse Proximal Policy Optimization is introduced, which applies the value loss to only a few well-separated states in each response to mitigate both ef...
Yi-Zhuo Li, Jian-Hao Yan, Yun Luo et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.