LoongReflect is proposed, a training framework that formulates reflection as a memory-control policy over a reversible trajectory tree using explicit reflect and backtrack actions and demonstrates consistent improvements over outcome-only reinforcement learning and self-distillation baselines.
Abstract
Large language model agents increasingly rely on long-horizon reasoning to solve complex tasks involving planning, tool use, and memory. A critical capability in such settings is reflection: assessing trajectory progress, identifying missing evidence and unreliable intermediate states, and deciding whether to continue, revise, or abandon the current branch. Learning effective reflection, however, is challenging because reflection is performed locally within the current branch, whereas its utility can only be determined by its contribution to the final trajectory outcome. This local-global mismatch makes outcome-based reinforcement learning provide only local, sparse and delayed supervision for reflective decisions. To solve these, we propose LoongReflect, a training framework that formulates reflection as a memory-control policy. The agent operates over a reversible trajectory tree using explicit reflect and backtrack actions. Reflection consolidates verified facts, missing evidence, and branch-specific risks into working memory, while backtracking removes an unreliable branch from the active context and preserves a concise corrective lesson. To learn this policy, LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism. A fast channel distills globally informed reflective behavior from a privileged teacher, with supervision restricted to reflection and backtracking tokens. A slow channel optimizes complete trajectories using outcome-based GRPO, aligning local control decisions with final task success. Experiments on multi-hop retrieval-augmented generation and mathematical reasoning benchmarks demonstrate consistent improvements over outcome-only reinforcement learning and self-distillation baselines.
Deep search requires LLM agents to decompose complex queries, search for evidence, and synthesize grounded answers, yet existing ReAct-style agents suffer from two limitations: role coupling, where one policy must handle planning, evidence use, and synthesis; and context accumulation, where growing search histories int...
Xing-Yu Wu, Yuchen Yan, Zhengxi Lu et al.· 0 citations
This work introduces SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask, which substantially improves best-of-N policy steering on real-world manipulation tasks.
Saksham Singh, Zhe-Yuan Hu, Max Sobol Mark et al.· arXiv.org· 2 citations
TIDE dynamically rebalanced teacher guidance and reward optimization should be dynamically rebalanced over training and jointly allocated across turns, and experiments support the effectiveness of TIDE's adaptive OPD--RL coordination.
APEx is proposed, a hierarchical experience utilization framework that organizes interaction history into instance-level trajectory memories and category-level procedural skills, and couples them through a closed-loop architecture of Executor, Distiller, and Planner, allowing ground-truth-free self-improvement with ski...
Jie Ding, Rui Sun, Xin-Yi Zhang et al.· 2 citations
Results support a qualified internalized-search reading: under the recipe the authors test, much of the measured RL gain corresponds to a change in sampling efficiency toward operating points the base model can already reach under search.
Wen-He Sun, Cun-Xiang Wang, Zi-Jun Yao et al.· 0 citations
Coding agents solve repository-level software engineering tasks through long trajectories of code inspection, search, editing, and testing. As a task progresses, earlier exploration becomes stale, so managing context is more than avoiding overflow: an agent must decide when to compact, what working state to preserve, a...
Xuan Zhang, Long-Tao Zheng, Cun-Xiao Du et al.· 7 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.