Skip to content

ASCT: Attentive Search over Counterfactual Trees for Credit Assignment in Agentic Reinforcement Learning

Sep 2026 · arXiv.org · Vol abs/2609.35215 · 0 citations · 25 references
Computer Science

TL;DR

Attentive Search over Counterfactual Trees (ASCT) is introduced, a framework that turns training-time multi-step search into local action credit and connects counterfactual evaluation to policy learning while deploying the actor alone.

Abstract

Terminal utility evaluates a complete agentic workflow, but learning requires credit for the decisions within it. We introduce Attentive Search over Counterfactual Trees (ASCT), a framework that turns training-time multi-step search into local action credit. At actor-visited states, an auxiliary tree evaluates alternative legal actions from the same recoverable prefix. Its action-value table is centered by the frozen actor's probabilities and supplies credit for PPO on actor-sampled trajectories. This protocol connects counterfactual evaluation to policy learning while deploying the actor alone. Uniform, UCT, and cost-aware AgentUCT instantiate the framework. On HotpotQA agentic retrieval-augmented generation, all three improve mean held-out utility over trajectory-return PPO and workflow-adapted VinePPO. Across three seeds, ASCT-AgentUCT reaches 0.6187 utility versus 0.5939 for VinePPO, with gains in answer F1 and execution cost, and uses 50.3% fewer recorded auxiliary Qwen tokens. Transfer and component-description studies examine the learned policies beyond the training setting.

View source

Similar papers

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

ProVer, a framework that targets potentially pivotal decisions for fine-grained credit assignment in agentic reinforcement learning, is introduced, highlighting the effectiveness and efficiency of selectively targeting pivotal decisions for fine-grained credit assignment in agentic reinforcement learning.

Dongwon Jung, H. Ramesh, Yi-Fan Wang et al. · 0 citations
Preprint Aug 2026

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence, is introduced and advances the understanding of credit assignment in multi-turn user interactions.

B. Ren, Yirong Mao, Yi Yang et al. · 1 citation

Learning What to Skip: Counterfactual Credit Assignment for Efficient Multi-Agent LLM Workflows

Learning What to Skip is introduced, which learns action-specific safety models from these interventions and combines held-out calibration with domain-native guards to select skips and reduces recorded token cost while matching or improving aggregate full-workflow accuracy in the evaluated settings.

Jin-Feng Xu, Zhe-Yu Chen, Zi-Yue Peng et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Matched-Context Group Policy Optimization for LLM Agent Training

Training long-horizon language model agents requires assigning credit to intermediate decisions from sparse terminal feedback. Within a finite group of sibling rollouts, useful action comparisons depend on their context, evidence for both alternatives, and how that evidence enters learning. We propose Matched-Context G...

Chen-Xu Zhang, Yao-Rui Shi, Wen-Long Zhang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.