Skip to content

Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training

Sep 2026 · arXiv.org · Vol abs/2609.34279 · 0 citations · 29 references
Computer Science

TL;DR

This work introduces DEO, which replaces challenger parameter updates with solver-guided task sampling, and shows that DEO learns distributionally robust reasoning ability.

Abstract

Self-evolving language models improve by generating tasks and learning from their own feedback, but adapting the task generator often requires a separate challenger-training loop. Can we generate tasks adapted to the current solver without explicitly training a challenger? We introduce \textbf{D}irect Self-\textbf{E}volving \textbf{O}ptimization (DEO), which replaces challenger parameter updates with solver-guided task sampling. The KL-regularized challenger objective defines an exponential tilt of a fixed base task distribution. DEO uses this distribution as a sampling target: a frozen LLM generates and mutates tasks, the solver scores them, and an approximate Metropolis selection rule refines the training pool. Only the solver is trained. Theoretically, for an idealized variant that samples exactly from the tilted distribution, and under regularity, local gradient-dominance, and initialization conditions, we show that DEO learns distributionally robust reasoning ability. In experiments, DEO achieves reasoning performance competitive with R-Zero while using over $50\%$ less wall-clock training time, and improves reasoning accuracy over a no-walk ablation. Replacing the task generator with a frozen API-only LLM further improves the local solver, illustrating a capability enabled by removing challenger training.

View source

Similar papers

#machine learning Preprint Oct 2026

Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards (RLVR) is reshaping how large language models (LLMs) acquire reasoning capabilities, yet the optimizers used remain largely inherited from teacher-forcing training. This raises a fundamental question: does RLVR induce a distinct optimization geometry that calls for a diffe...

Zhi-Peng Chen, Wayne Xin Zhao, Ji-Rong Wen · 0 citations
#artificial intelligence Preprint Oct 2026

Selection Is Not Learning: Choosing Model Futures

Self-evolving language models train on tasks they generate, so every round must decide which updated model the next round inherits. Current loops such as R-Zero, Agent0, and Skill Self-Play make this decision through the task, admitting tasks that look learnable and inheriting every resulting update. We identify a sele...

Hao-Yi Yang, Kshitij Mishra, Nils Lukas et al. · 0 citations

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation

When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain

A holistic framework based on learnable information gain, which measures how much novel, parameterizable information a round provides relative to the previous round, and proposes ATRI (Adaptive Training Regulation via Information-gain), which reweights samples within a round and halts training across rounds when inform...

Chen-Xu Wang, Chao-Zhuo Li, Xin-Ze Shi et al. · 0 citations

Self-Play Pretraining with Zero Data

Self-Play Pretraining with Zero Data is introduced, an initial proof-of-concept towards realizing a more general approach to pretraining, and whether zero-shot performance on natural data improves predictably with self-play compute is tested.

Aditya Cowsik, Kfir Dolev, Michael Y. Li et al. · 3 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.