Attentive Search over Counterfactual Trees (ASCT) is introduced, a framework that turns training-time multi-step search into local action credit and connects counterfactual evaluation to policy learning while deploying the actor alone.
Abstract
Terminal utility evaluates a complete agentic workflow, but learning requires credit for the decisions within it. We introduce Attentive Search over Counterfactual Trees (ASCT), a framework that turns training-time multi-step search into local action credit. At actor-visited states, an auxiliary tree evaluates alternative legal actions from the same recoverable prefix. Its action-value table is centered by the frozen actor's probabilities and supplies credit for PPO on actor-sampled trajectories. This protocol connects counterfactual evaluation to policy learning while deploying the actor alone. Uniform, UCT, and cost-aware AgentUCT instantiate the framework. On HotpotQA agentic retrieval-augmented generation, all three improve mean held-out utility over trajectory-return PPO and workflow-adapted VinePPO. Across three seeds, ASCT-AgentUCT reaches 0.6187 utility versus 0.5939 for VinePPO, with gains in answer F1 and execution cost, and uses 50.3% fewer recorded auxiliary Qwen tokens. Transfer and component-description studies examine the learned policies beyond the training setting.
ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment in agentic reinforcement learning, is introduced, highlighting the effectiveness and efficiency of selectively targeting pivotal decisions for fine-grained credit assignment in agentic reinforcement learning.
Dongwon Jung, H. Ramesh, Yi-Fan Wang et al.· arXiv.org· 0 citations
Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence, is introduced and advances the understanding of credit assignment in multi-turn user interactions.
It is argued that the right resolution is state-dependent, and GACA, a critic-free estimator whose granularity follows an uncertainty-based criticality proxy is proposed, improves task success over GRPO and GiGPO at both 1.5B and 7B scales.
Tao Liang, Yang Liu, Shang Luo et al.· 3 citations
Learning What to Skip is introduced, which learns action-specific safety models from these interventions and combines held-out calibration with domain-native guards to select skips and reduces recorded token cost while matching or improving aggregate full-workflow accuracy in the evaluated settings.
Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context G...
Chen-Xu Zhang, Yao-Rui Shi, Wen-Long Zhang et al.· 0 citations
These results support accounting for selection history when constructing and evaluating search-agent rollouts and rank first on seven QA benchmarks using Qwen3-4B, Qwen3-8B, and Qwen2.5-7B.
Zeng-Huang Fu, Ning Chen, Ming-Da Jia et al.· arXiv.org· 1 citation