Skip to content

TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment

Sep 2026 · Conference on Empirical Methods in Natural Language Processing · Vol abs/2609.33426 · 0 citations · 39 references
Computer Science

TL;DR

Teacher Alignment is proposed, which directly adapts the teacher toward the student's distribution without discarding data or degrading reasoning quality, and which significantly outperforms baselines across diverse reasoning benchmarks and distillation methods.

Abstract

Reasoning distillation from powerful teacher models to smaller students faces the Gap Curse: as teachers grow more sophisticated, their complex distributions increasingly diverge from what students can approximate, causing performance degradation. Existing mitigation strategies either filter out challenging examples through data selection or introduce weaker intermediate assistant models, inherently compromising supervision coverage or quality. We propose Teacher Alignment, which directly adapts the teacher toward the student's distribution without discarding data or degrading reasoning quality. However, naive alignment through standard knowledge distillation triggers catastrophic collapse of the teacher's reasoning capabilities. To address this, we reformulate teacher alignment as reinforcement learning and introduce TeacherGRPO, built on Group Relative Policy Optimization with two key innovations: (i) Curriculum Selective Alignment applies dual token- and distribution-level curricula to focus rewards on high-signal reasoning gaps while filtering noise from trivial tokens and uncertain tail distributions, and (ii) Importance-Adaptive Length Regularization selectively penalizes verbose redundancy while preserving pedagogically critical reasoning steps. The aligned teacher then distills knowledge to students via standard pipelines. Extensive experiments show TeacherGRPO significantly outperforms baselines across diverse reasoning benchmarks and distillation methods. Our code is available at https://github.com/LzyFischer/TeacherGRPO.

Read PDF

Similar papers

Preprint Aug 2026

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

This work proposes Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward.

Chen Yang, Hai-Yuan Wan, Rengrong Xiong et al. · 2 citations

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

CaRE-KD is proposed, a confidence-gated distillation framework that replaces static objectives with uncertainty-adaptive optimization and provides a gradient-level analysis showing how this dual-granularity design induces a conditional calibration mechanism that prior static divergences cannot reproduce.

Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty · 0 citations
#artificial intelligence Preprint Oct 2026

Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers

Reinforcement learning can substantially improve a reasoning teacher, but it is unclear which of those improvements survive when the teacher supervises a smaller on-policy student. We study this question in mathematical reasoning by comparing teacher lineages before and after GRPO, multiple student scales, direct GRPO,...

Xiao-Yu Chen, Bo Shao, Tian-Gang Zhu et al. · 0 citations
#natural language process... Preprint Aug 2026

SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning

This work introduces SPEAR (Symbolic Process Evaluation and Alignment Reward), a training-free and plug-and-play process reward method for sequence-level on-policy distillation that effectively bridges the reasoning gap between student and teacher models via sequence-level distillation with efficient dense process rewa...

Zhuo-Chun Li, Yuelyu Ji, Yiming Zeng et al. · 1 citation

Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Adaptive On-Policy Self-Distillation is proposed, which adapts what information the teacher receives and how strongly its feedback influences learning, and encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI...

Jia-Cheng Du, Wei-Wei Xie, Tian-Yi Du et al. · 0 citations

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation, shows that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

Safaeid Hossain Arib, Rabeya Akter, I. N. Swapnil et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.