Skip to content

OPSRD: On-Policy Self-Role Distillation

Sep 2026 · arXiv.org · Vol abs/2609.39884 · 0 citations · 25 references
Computer Science

TL;DR

OPSRD is introduced, which uses a fixed expert role as privileged teaching context for on-policy self-distillation without reference solutions for on-policy self-distillation without reference solutions.

Abstract

Role prompting elicits specialized behavior from large language models through an expert identity, offering a lightweight way to guide reasoning on demanding tasks. However, evaluating or distilling complete role-prompted answers can miss useful next-token preferences when the sampled solution remains incorrect. Transferring these preferences also requires an objective that reaches alternatives the student rarely predicts. We introduce OPSRD, which uses a fixed expert role as privileged teaching context for on-policy self-distillation without reference solutions. A role-free student generates a trajectory, and a frozen instance of the same base model supplies role-conditioned distributions on its exact prefixes, exposing alternatives beyond the sampled continuation. Teacher-weighted forward KL targets alternatives the student underestimates, with clipping to limit individual vocabulary contributions. Supervision is restricted to the highest-entropy half of student positions, concentrating learning where predictions are uncertain. Experiments on three competition-math benchmarks with Qwen3-1.7B, 4B, and 8B show improvements over the base models without role prompts at inference. Forward KL achieves the highest macro-averaged accuracy among the three evaluated divergences at every scale. Code is available at https://github.com/zhansan114514/OPSRD.

View source

Similar papers

#natural language process... Preprint Oct 2026

Self-Distillation with Verbal Regularization

On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distribut...

Jiazheng Li, Yawei Li, Jun-Ru Lu et al. · 0 citations

Learning from Think-Mode Advantage via On-Policy Distillation

ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison.

Wan-Qi Ren, Jian-Xiang Wang, Danyang Liu et al. · 0 citations
#machine learning Preprint Oct 2026

One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation

On-policy self-distillation (OPSD) improves reasoning by training a model on its own trajectories with dense token-level supervision from a privileged copy conditioned on a reference solution. Existing methods typically use a single teacher distribution conditioned on the complete reference. We identify a limitation of...

Khouloud Saadi, Di Wang · 0 citations

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation, shows that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

Safaeid Hossain Arib, Rabeya Akter, I. N. Swapnil et al. · 1 citation
#machine learning Preprint Oct 2026

Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

This work systematically evaluates 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode and finds that PI effectiveness depends on its content, generation source, and the student model.

Kai-Jian Zou, Emiliano Penaloza, MohammadHossein Rezaei et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.