Skip to content

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

Sep 2026 · arXiv.org · Vol abs/2609.33200 · 1 citation · 57 references
Computer Science

TL;DR

On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation, shows that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

Abstract

On-policy self-distillation trains reasoning models on their own trajectories using dense token distribution guidance from a privileged teacher with access to a verified solution. This supervision transfers what the teacher predicts without directly transferring where it attends within the preceding context. We introduce On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation. Because the privileged teacher can attend to verified solution tokens unavailable to the student, OPASD projects teacher attention onto student-visible positions and renormalizes the resulting distribution before alignment. Across three model sizes and four competition-level mathematics benchmarks, OPASD consistently outperforms token-only OPSD, improving average accuracy by 4.98 to 8.40 percentage points. OPASD also avoids the response-length inflation and performance degradation observed with token-only distillation, reducing generated rollout tokens by 73.9% and estimated model compute by 72.6% while training 1.53x faster. These results show that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

View source

Similar papers

Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Adaptive On-Policy Self-Distillation is proposed, which adapts what information the teacher receives and how strongly its feedback influences learning, and encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI...

Jia-Cheng Du, Wei-Wei Xie, Tian-Yi Du et al. · 0 citations

Recursive Self-Improvement via On-Policy Distillation for Reasoning

DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks, and its comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks.

Shang-Jian Yin, Ze-Hao Zhao, Kavosh Asadi et al. · 6 citations

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation

Learning from Think-Mode Advantage via On-Policy Distillation

ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison.

Wan-Qi Ren, Jian-Xiang Wang, Dan-Xuan Liu et al. · 0 citations
Preprint Sep 2026

CA-OPD: Confidence-Aware On-Policy Distillation for Structured Visual Prediction

Evaluated in a multi-teacher setting for GUI grounding and optical character recognition, CA-OPD substantially improves the Qwen3.5-0.8B baseline across all six target benchmarks, including gains of $9.50$ points on ScreenSpot-Pro and OCRBench-v2 English.

Meng-Hao Li, Lin-Jie Mu, Yin Wang et al. · 2 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.