Skip to content

Data-free On-policy Distillation

Sep 2026 · 1 citation · 61 references
Computer Science

TL;DR

DF-OPD matches and even surpasses real data, and the questions it produces track the teacher's own post-training data on three key diagnostics of training dynamics, which other real datasets do not.

Abstract

On-policy distillation (OPD) has become a standard component of frontier post-training pipelines, yet how much its training data actually contributes has gone largely unexamined. On the two teacher--student pairings most common in practice, we find OPD almost indifferent to its data: eight prompts already match a 17k-problem dataset, and three independently built datasets whose difficulty and teacher--student KL differ several-fold produce nearly indistinguishable training curves. Two causes account for this. First, the unit of data in OPD is the state a prompt leads to, not the prompt itself: a single prompt keeps exposing new teacher correction as sampling continues, while the marginal value of additional prompts collapses after eight. Second, replacing mathematics with competitive programming still recovers over ninety percent of the in-domain gain, indicating that OPD transfers the teacher's mode of reasoning rather than knowledge related to the data. We take this to its limit with \textbf{Data-free On-policy Distillation} (DF-OPD), in which the teacher writes its own training questions under a simple prompt---no external data, no quality filtering---leaving a system of just two policies. DF-OPD matches and even surpasses real data, and the questions it produces track the teacher's own post-training data on three key diagnostics of training dynamics, which other real datasets do not. Applied to multi-teacher distillation, where the (prompt, domain) pairs normally have to be derived from post-training data that is often out of reach, 1k self-generated questions close 98.5\% of the available headroom, even surpassing the 96.6\% reached with 7k real examples. Together these results invite a reassessment of the role data plays in OPD.

View source

Similar papers

CLOOPD: Closing the Learner Loop in On-Policy Distillation

Across six 300-step runs on an 8-H20 node, every CLOOPD policy improves the one-pass TOP-D anchor at comparable teacher-token scale: macro accuracy rises from 15.41 to 17.78 with CLOOPD-Fixed2 and 19.36 with CLOOPD-Fixed3.

Keye Zheng, Han-Yu Li, Zhan Cheng et al. · 0 citations
Preprint Aug 2026

WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training

WDL-OPD is introduced, a mixture-constrained co-training method with two trainable policies that shows that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$^2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.

Ze-Hao Chen, Gong-Xun Li, Tianxiang Ai et al. · 0 citations
#artificial intelligence Preprint Sep 2026

What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection

This paper presents an empirical study of data efficiency and data selection in OPD, and proposes a simple data selection method that selects only hard examples for training, where even"unsolvable"examples that completely exceed the teacher's capability can be successfully used.

Zhi-Nan Hou, Jiaqi Zhang, Xun-Liang Cai et al. · 6 citations

ReTaCo: Residual-Target Control for On-Policy Distillation

On-policy distillation (OPD) trains a student on its own generated prefixes with token-level teacher feedback, but transmitting or storing the teacher's full-vocabulary distribution at every token is costly. Entropy-aware OPD (EOPD) adds forward supervision to reverse KL to help the student recover plausible tokens it...

Zi-Xiang Ni, Zhuo-Lun Hu, Ren-Jie Cao et al. · 0 citations
#machine learning Preprint Sep 2026

LastOPD: Taming Collapse in Latent On-Policy Distillation

LastOPD is proposed, which applies the latent signal only at the last-layer state, the common interface both LM heads read, and only during a 10-step crossfade into token-level OPD, which keeps the useful part of the latent signal and hands the student to token-level supervision before the collapse sets in.

Jie Yang, Zheng-Yu Fang, Ze-Lin Xu et al. · 3 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.