Skip to content

Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents

Sep 2026 · arXiv.org · Vol abs/2609.08919 · 1 citation · 24 references
Computer Science

TL;DR

Experiments across diverse agent benchmarks show that \textit{Experience Funnel} consistently improves agent capability over state-only evolution and policy-internalization approaches, while progressively converting useful explicit experience into autonomous policy competence.

Abstract

Autonomous agents powered by large language models (LLMs) continuously accumulate experience through interaction, creating an opportunity to improve future behavior through self-evolution. A fundamental challenge is how to transform abundant, task-specific interaction experience into reusable model competence without sacrificing the ability to adapt rapidly to newly observed evidence. Explicit textual states, such as skills and agent harnesses, provide fast, human-readable and editable adaptation, but incur persistent dependence on external context; parametric policies provide compact and reusable competence, but are substantially slower to update. We present \textit{Experience Funnel}, a self-evolving framework that couples fast state adaptation with slow policy consolidation in an alternating loop. Interaction trajectories are first distilled into an explicit textual state, where newly acquired experience can be rapidly incorporated and validated. The framework then selectively identifies state-enabled behavior that remains useful across state revisions and consolidates it into the policy through transition-aware distillation. The updated state--policy pair subsequently generates new rollouts, providing fresh evidence for the next round of state adaptation and policy consolidation. Experiments across diverse agent benchmarks show that \textit{Experience Funnel} consistently improves agent capability over state-only evolution and policy-internalization approaches, while progressively converting useful explicit experience into autonomous policy competence.

View source

Similar papers

Preprint Oct 2026

Darwin-Claw: Environment-Adaptive Experience Consolidation for Recursively Self-Evolving Agents

While Large Language Model (LLM) agents have demonstrated remarkable capabilities across complex tasks, they predominantly operate in isolated sessions, lacking the ability to continually learn from historical interactions or adapt to user-specific environments. Existing self-evolution paradigms often indiscriminately...

Ming-Ze Yin, Xiao-Han Wang, Dian Li et al. · 0 citations
Preprint Oct 2026

Productive State-Space Exploration for Multi-turn RL in LLM Agents

Reinforcement Learning (RL) for Large Language Model (LLM) agents in multi-turn, long-horizon environments remains challenging due to inefficient action selection and repetitive behavioral patterns. In particular, agents often become trapped in high-confidence yet unproductive behaviors, repeatedly executing invalid ac...

Mohammed Haroon Dupty, Yong-Liang Goh, Song Guo et al. · 0 citations

Beyond Skill Evolution: Self-Evolving Context Management Policies for Long-Horizon Agent Harnesses

ContextEvo, a framework that learns a context policy from long-horizon trajectories and reconstructs the model-visible context at key decision points, identifies context-related failures, and applies targeted policy updates, is introduced.

Wei-Yuan Li, Jing-Heng Xu, Ai-Li Chen et al. · 0 citations

RoboFoundry: System-as-Policy Evolution for Self-Learning Embodied Agents

RoboFoundry is proposed, the first embodied agentic framework that formulates this process of system-as-policy evolution across foundation models as Self-Evolving System-as-Policy, highlighting its potential for fully autonomous embodied agents.

Jing-Song Liang, Shu-Hao Liao, Shi-Zhe Zhang et al. · 5 citations · ⚡2

The Right Lesson at the Right Step: Deriving Control Updates for Self-Evolving Agents

Results show that self-evolving agents should place experience inside the control flow, rather than only store it as text, and show how learned conventions must reach the right execution step on long tool-use environments where learned conventions must reach the right execution step.

Yun-He Su, Zi-Yi Dong, Tong Yu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.