Skip to content

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

Sep 2026 · 0 citations · 25 references
Computer Science

TL;DR

Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization at the level of environment-facing segments that outperforms existing baselines on the ALFWorld and WebShop benchmarks.

Abstract

Agentic reinforcement learning (RL) trains a large language model (LLM) to act over long, multi-step interactions. However, a single localized error can cause task failure, while trajectory-level rewards provide limited guidance for assigning credit to individual decisions. To address this limitation, we introduce Segment-level Hindsight Advantage Reweighting for Policy Optimization (SHARPO), a credit-assignment mechanism that refines Group Relative Policy Optimization (GRPO) at the level of environment-facing segments. Inspired by the existing on-policy self-distillation (OPSD) method, SHARPO computes teacher-student log-probability gaps within each segment and uses the resulting signal to compute a bounded multiplier on the GRPO advantage. This multiplier is shared by all tokens within the segment, allowing credit to vary across different segments. With Qwen2.5-7B-Instruct, SHARPO outperforms existing baselines on the ALFWorld and WebShop benchmarks, including GRPO, SDAR, RLSD, and StepOPSD.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

LLM Value Functions for Credit Assignment in Agentic RL

LLM Value Function RL is introduced, which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic, and consistently and substantially outperforms both GRPO and PPO.

Kai-Jie Zhu, Wei-Xiang Yan, Omar Attia et al. · 1 citation
#machine learning Preprint Sep 2026

Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL

Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and adherence to task re...

Jin-Hao Dong, Liang Zhao, Zi-Hao Yue et al. · 1 citation
Preprint Nov 2026

BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates, and accumulates context-relative returns in a global memory with balanced aggregation and reliability weighting to control bias and noise from heterogeneous tasks.

Xuan-Ren Chen, Chong-Yang Tao, Wen-Peng Hu et al. · 0 citations

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

This work proposes self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit, and demonstrates that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

Zhenrui Yue, Hui-Min Zeng, Yue-Qi Wang et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.