HindSearch is introduced, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions.
Abstract
Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge's access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement.
This work proposes AHEAD, a step-aware framework that matches different supervision sources to different step types and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
Xiaolong Jin, Dingmin Wang, Vijay Lingam et al.· 1 citation
Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them.
Hao-Yu Zheng, Yun Zhu, Qing Wang et al.· 1 citation
Group-relative methods for reinforcement learning with verifiable rewards (RLVR) learn from differences in rollout outcomes. Independently sampling complete trajectories is costly and does not explicitly explore the decision space at critical positions. Feedback on a completed trajectory can reveal which earlier choice...
Fan-Chao Chen, Heng-Yu Fu, Shivaram Venkataraman et al.· 0 citations
Evidence Anchors are constructed, which are concise, step-level evidence snippets extracted from the web, as privileged information that captures key reasoning steps without revealing the entire answer path, and SSPO, which converts teacher-student disagreement into step-level advantage weights within GRPO, applied exc...
Haoze Wu, Chu-Qiao Kuang, Tian-Yi Zhuang et al.· 1 citation
A natural extension to preference-conditioned multi-objective RL (MORL) relabels transitions with the preference direction the agent achieved rather than the one asked for, and a single-parameter convex combination pulling the achieved direction back towards the requested preference is introduced.
Baptiste Bonin, Caro Strickland, Audrey Durand· 0 citations
Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. Invisible Advanta...
Mu-Yang Li, Jie Yang, Zheng-Yu Fang et al.· 1 citation· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
