Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.

Cross-Benchmark Generalization in Long-Horizon Agents

Desc descriptive evidence is provided that long-horizon multi-tool post-training can change ways of working that transfer beyond its training domain, and both software-engineering benchmarks improve despite the training collection containing no software-engineering tasks.

Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

Self-Review Reinforcement Learning consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.

Knowing When Not to Reuse: Conditional Experience Transfer in Autonomous LLM Post-Training

Large language models offer broad capabilities, but adapting them to evolving domains, tools, and requirements often entails repeated post-training. Autonomous systems automate parts of this process by proposing updates, training candidates, and using evaluation feedback to select subsequent proposals. As evidence accumulates, a central problem emerges: which past update evidence remains actionable after subsequent training has changed the parent model? An update's effect depends on its parent, data, and training stage. Treating past success as context-free permission can waste compute. If the resulting child is promoted, it can also degrade the subsequent training trajectory. We formulate this problem as conditional experience transfer and introduce Boundary-Calibrated Intervention Transfer (BCIT), a method that authorizes experience reuse before weight-changing training. BCIT binds an observed effect to its source context, checks applicability conditions, vetoes candidates with named hard conflicts, and obtains current-state evidence through a bounded training trial when needed. Fully trained candidates still face a shared adoption rule, and only observed events extend memory. On one 4B model adapted across finance reasoning, text-to-SQL, and function calling, candidate updates exhibit heterogeneous target and retention effects across the evaluated contexts. Under matched candidates, evidence, and compute, BCIT authorizes fewer harmful updates and attains higher equal-budget final-model quality than the evaluated alternatives. These results support treating experience authorization as a distinct problem in autonomous post-training.

OLIVIA: Online Learning via Inference-time Action Adaptation

Multi-Turn Reinforcement Learning for Large Language Models: From Theory to Practice with Amazon SageMaker AI

Production large language model (LLM) based systems such as coding agents, web navigators, and tool-calling assistants operate over multiple turns of interaction with users, tools, and environments. Pretrained LLMs, depending on their size, can either underperform in these settings due to misalignment with the system's interaction mechanics, or, when capable, incur prohibitive latency. Fine-tuning right-sized models addresses both accuracy and latency, but training such multi-turn agents requires Reinforcement Learning (RL), where the model acts as a policy optimizing long-horizon outcomes across sequential interactions. This poses challenges absent from single-turn settings: credit assignment over long trajectories, reward design for sparse and delayed feedback, state and context management as observation histories grow, environment scaling for parallel rollout collection, and training stability under prompt/environment distribution shift. This hands-on problem-solving tutorial provides both a rigorous algorithmic and practical introduction to multi-turn RL finetuning for LLMs. Using Amazon SageMaker AI, participants progress through four labs: (1) environment and reward function design, (2) multi-turn trajectory collection and Group Relative Policy Optimization (GRPO)-based training, (3) reward densification and credit assignment strategies, and (4) evaluation, failure diagnosis and deployment. We cover state-of-the-art multi-turn RL finetuning algorithms, turn-level vs. trajectory-level reward design, and production grade monitoring for reward hacking detection. The tutorial targets machine learning (ML) engineers, data scientists, and researchers who build agentic LLM systems. No prior RL experience is required. All materials will be publicly available on GitHub.