ODYSSE is presented, a Reinforced Fine-Tuning (RFT) framework for personalized agentic reasoning designed to address long action horizons and strong cross-step dependencies in personalized agentic reasoning, and an episodic batch sampler that groups actions from the same episode into unified training batches, facilitating coherent optimization under ESPO.
Abstract
Agentic systems have rapidly advanced in their ability to interact with real-world environments, leverage external tools, and provide services for users. However, unlike natural-world tasks that assume well-defined instructions, human-centered scenarios are characterized by ambiguous requests that lead to large, open-ended solution spaces. Decoding users'personalized preferences is therefore essential for narrowing the candidate solution space. This introduces a new challenge, personalized agentic reasoning, which requires agents to jointly interact with both users and environments to deliver personalized services. In this paper, we present ODYSSE, a Reinforced Fine-Tuning (RFT) framework for personalized agentic reasoning. At its core, ODYSSE proposes Episode-wise GRPO (ESPO), a novel extension of Group Relative Policy Optimization (GRPO) designed to address long action horizons and strong cross-step dependencies in personalized agentic reasoning. Rather than optimizing individual steps independently, ESPO introduces an episode-level reward mechanism together with episodic advantage estimation, enabling upstream evidence to effectively guide downstream personalized decisions and allowing agents to progressively resolve ambiguous user requests across multiple interaction steps. We further propose an episodic batch sampler that groups actions from the same episode into unified training batches, facilitating coherent optimization under ESPO. We evaluate ODYSSE on realistic long-horizon personalized GUI reasoning tasks. Experimental results demonstrate that ODYSSE consistently outperforms both specialist and general-purpose LVLMs, highlighting its effectiveness for personalized agentic reasoning.
Group Planning-aware Policy Optimization (PlanPO) is proposed, a simple yet effective RL method for learning generalizable planning abilities beyond task-specific high-quality behavior patterns that enables agents to actively learn generalizable and deliberate behaviors spanning interaction planning and textual generation from high-quality rollouts, without degenerating into vanilla length minimization.
D. Liang, Liyuan He, Xuan Feng et al.· 0 citations
While LLM agents demonstrate strong reasoning abilities in compact and well-defined scenarios, they struggle to maintain robustness and effectiveness when faced with large-scale, diverse, and dynamic real-world environments that demand seamless tool integration. To address this gap, we introduce ToolVerse, a comprehensive framework that scales up agentic RL environments and enables agents to perform complex long-horizon reasoning in Tool-Integrated Reasoning (TIR) tasks. First, ToolVerse automatically builds the massive executable agent training environments from nearly 400 real-world Model Context Protocols (MCPs) that contain about 4500 tools. Second, we propose a task design strategy based on a tool dependency graph, utilizing Dynamic Unlocking Sampling Algorithm to generate long-horizon tasks, and produce GUST (Graph Unlocking Sampling Tasks) dataset. Third, to alleviate the credit assigment problem in long-horizon agentic RL, we propose a fine-grained Turn-Aware Relative Advantage algorithm. We conduct extensive Agentic RL training using ToolVerse and evaluate our framework on serveral agentic benchmarks. Experimental results demonstrate that our framework significantly strengthens LLMs'capabilities in long-horizon tool use, achieving a marked performance boost and showcasing robust reasoning within dynamic environments.
Shuaiyu Zhou, Fengpeng Yue, Zengjie Hu et al.· 2 citations
This research bridges theoretical foundations of reinforcement learning and graph-based memory with autonomous agent workflows, and offers a practical, scalable reference framework for developing artificial intelligence technologies in complex, multi-step autonomous systems.
River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization is proposed, which achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks.
Yi-Fan Yao, Bo Pang, Xuan-Phi Nguyen et al.· 0 citations
This work proposes a reinforcement learning-based framework for fine-tuning and optimizing Small Language Models (SLMs) used in Agentic AI systems, with a focus on improving tool invocation and decision-making over complex, multi-step tasks. The framework targets both open-source and proprietary language models operating in interactive environments that integrate multiple data sources and external APIs through autonomous agents. We leverage Grouped Relative Policy Optimization (GRPO) to generate and evaluate multiple complete execution trajectories per query within a simulated interaction environment, assigning trajectory-level rewards through an autonomous LLM-based evaluation module (RULER). High-reward trajectories are used to iteratively update SLM parameters, producing a compact, specialized policy that improves tool selection accuracy, response quality, and robustness under dynamic query distributions. To address the limitations of fine-tuning proprietary models, we introduce a memory-based middleware that stores query-trajectory-reward tuples and reuses high-reward trajectories at inference time for semantically similar queries, enabling trajectory-level adaptation without modifying underlying model parameters. A learned reuse policy governs whether to replay, adapt, or regenerate trajectories, yielding substantial reductions in inference latency. Overall, the proposed architecture combines GRPO-based policy refinement with learned trajectory reuse, improving accuracy and efficiency while reducing reliance on expensive large-model retraining and preserving deployment flexibility across Agentic AI use cases such as database querying, web search, and multi-tool orchestration. We further validate the framework on a multi-agent information-seeking task involving enterprise email search (Enron corpus) with dynamic redirection to web search via SearXNG, demonstrating generalization beyond incident-management workflows.
Subhadip Mandal, Shalini Mittal, Sukanya Bag et al.· 2026 6th International Confe...· 0 citations
This work introduces AgentMercury, a scalable framework for synthesizing executable environments from high-level business scenarios that improves substantially on both enterprise workflows and out-of-domain benchmarks spanning reasoning, coding, scientific computing, and tool use.