Skip to content
Preprint

Asynchronous On-Policy Self-Distillation

Oct 2026 · 0 citations · 34 references
Computer Science

TL;DR

This work introduces Asynchronous On-Policy Self-Distillation (AsyncOPSD), a framework that decouples rollout generation from learner updates to improve training throughput and introduces a distillation objective based on staleness-aware policy control and token-level entropy weighting.

Abstract

Unlike on-policy distillation (OPD), which depends on a separate teacher model, on-policy self-distillation (OPSD) leverages a single language model acting simultaneously as the student and the teacher, with the teacher conditioned on privileged information. Despite this architectural simplicity, synchronous rollout generation dominates overall training time, creating a severe systems bottleneck when scaling OPSD. To address this, we introduce Asynchronous On-Policy Self-Distillation (AsyncOPSD), a framework that decouples rollout generation from learner updates to improve training throughput. Since the teacher and student share parameters, AsyncOPSD eliminates the multi-model coordination overhead inherent to OPD architectures in asynchronous training. As the student policy evolves, buffered trajectories generated by older policies become stale, potentially degrading performance. To mitigate this, we introduce a distillation objective based on staleness-aware policy control and token-level entropy weighting. Across three model sizes and mathematical reasoning benchmarks, AsyncOPSD improves the training throughput by $1.96$--$2.25\times$ over synchronous OPSD while increasing average benchmark accuracy.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

On-Policy Self-Distillation with Teacher Evolution

This paper proposes Teacher-Evolution Self-Distillation (TESD), which enables the self-teacher to generate rollouts and receive environmental feedback based on task correctness and context independence and performs teacher-oriented optimization to continuously enhance the model's self-teaching capability during distill...

Shu-Fan Shen, Zhong-Ni Hou, Chao Bi et al. · 0 citations
Preprint Nov 2026

Bootstrapped On-Policy Distillation

On-policy distillation (OPD) improves language models by distilling teacher predictions on student-generated trajectories, while on-policy self-distillation (OPSD) removes the need for an external teacher by using a reference-conditioned copy of the model as the teacher. Despite its great early improvements, we find th...

Yue Wu, Xue-Yu Zhou, Yang Hu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Proposing Better Rollouts for On-Policy Distillation

On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the cu...

Si-Rui Chen, Yun-Zhe Qi, Hao-Bo Xu et al. · 0 citations
Preprint Oct 2026

Bridged Self-Distillation for On-Policy Learning

On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavaila...

Run-Sheng Chen, Yihong Liu, Rao-Yuan Zhao et al. · 0 citations
#artificial intelligence Preprint Oct 2026

DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision

On-policy distillation (OPD) provides dense token-level supervision on student rollouts, but this supervision can become unreliable when the student and teacher diverge, either because of a large initial policy gap or compounding errors in long-horizon reasoning and tool use. We introduce Divergence-Triggered Dynamic D...

Qi-Jia He, Ze-Qun Li, S.-S. Luo et al. · 0 citations

Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solution. We separate these roles in a factor...

Md. Ismail Hossain, Humaira Kousar, Isidora Chara Tourni · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.