Skip to content

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Sep 2026 · 5 citations · ⚡ 1 influential · 75 references
Computer Science

TL;DR

Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.

Abstract

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose \textbf{RISE} (\textbf{R}ecursive \textbf{I}mprovement via \textbf{S}elf-\textbf{E}xtrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.

View source

Similar papers

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

On-Policy Attention Self-Distillation (OPASD), which complements token-level supervision with solution-conditioned attention distillation, shows that solution-conditioned attention provides a complementary supervision signal that makes on-policy self-distillation more accurate, stable, and compute-efficient.

Safaeid Hossain Arib, Rabeya Akter, I. N. Swapnil et al. · 1 citation

Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Adaptive On-Policy Self-Distillation is proposed, which adapts what information the teacher receives and how strongly its feedback influences learning, and encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI...

Jia-Cheng Du, Wei-Wei Xie, Tian-Yi Du et al. · 0 citations
Preprint Aug 2026

REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation

REOPD combines a token-level compatibility weight with a batch-level adaptive budget, yielding a token-wise coefficient $\lambda_{b,t}=1+\gamma_b q_t$ that preserves teacher alignment while selectively extrapolating along reliable teacher-reference directions, demonstrating effective fine-grained reliability adaptation...

Yang Sun, Li-Chao Ma, Houyuan Qin et al. · 3 citations

Learning from Think-Mode Advantage via On-Policy Distillation

ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison.

Wan-Qi Ren, Jian-Xiang Wang, Danyang Liu et al. · 0 citations
Preprint Aug 2026

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Self-OPD is introduced, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision and outperforms prior RL and OPD methods without task-specific teachers.

Shi-Yi Zhang, Mu-Shui Liu, Yunze Tong et al. · 1 citation

Related blog posts

Hugging Face Blog Nov 10, 2026

Decision Models in Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.