Skip to content
Preprint

Self-Improving Large Language Models via Progressive Experience Evolution

Aug 2026 · 2 citations · 26 references
Computer Science

TL;DR

SPEE is a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization, and demonstrates that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales.

Abstract

Large language models (LLMs) capable of self-improvement require not only effective policy optimization, but also a principled mechanism for transforming transient interaction experience into persistent model capabilities. Existing self-improvement paradigms remain fragmented: test-time methods can explicitly extract experience but cannot internalize it into model parameters, whereas training-time optimization methods can update model parameters but lack an explicit mechanism for accumulating transferable experience. Bridging these two paradigms requires a critical intermediate stage that remains underexplored, namely \emph{experience distillation}. To address this gap, we propose \textbf{SPEE} (\textbf{S}elf-\textbf{P}rogressive \textbf{E}xperience \textbf{E}volution), a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization. During explicit experience evolution, SPEE reflects on trajectories collected from multiple interactions to extract, verify, and progressively evolve transferable experience, which is subsequently internalized into the policy through privilege-guided On-Policy Self-Distillation (OPSD). During implicit policy optimization, reward-driven reinforcement learning leverages these internalized priors to explore novel solution strategies. In the experience evolution stage, a continuously evolving global experience pool consolidates knowledge from both successful and failed trajectories, filters out low-utility experience, and mitigates post-hoc rationalization induced by individual trajectories. Experiments on five mathematical reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales. The source code is available at https://github.com/rrrsj/SPEE.

View source

Similar papers

#natural language process... Preprint Aug 2026

S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies, and provide a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable se...

Jia-Jun Shi, Siyang Tao, Yu-Hao Wu et al. · 4 citations

Rep2Skill: Representation-Guided Skill Self-Evolution for LLM Agents

Textual skills enable large language model (LLM) based agents to accumulate reusable procedural knowledge without updating model parameters. Yet existing skill evolution remains largely confined to the text space: an optimizer must diagnose success and failure patterns, and revise skills solely from long execution traj...

Kai-Xin Zhang, Chang-Ming Li, Ying-Dong Shi et al. · 0 citations

Towards Evolving Context Parameterization for Large Language Models

This work proposed PLUME, a training-free method that constructs a global update representation, activates memory evidence to form a local parameter view, and adaptively integrates their predictions during decoding, which demonstrated its effectiveness in sequential evolution settings.

Xiao Shi, Zhe-Rui Li, Yi-Ming Jiang et al. · 0 citations

Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents

Experiments across diverse agent benchmarks show that \textit{Experience Funnel} consistently improves agent capability over state-only evolution and policy-internalization approaches, while progressively converting useful explicit experience into autonomous policy competence.

Wen-Bo Gao, Zhao-Mou Song, Zhi-Yuan Ji et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.