Skip to content

Dense Is Not Enough: Hierarchical Supervision Allocation for Long-Horizon On-Policy Distillation

Sep 2026 · 0 citations · 35 references
Computer Science

TL;DR

This work forms long-horizon OPD as hierarchical supervision allocation as well as a coarse-to-fine framework that organizes supervision through Locate, Validate, and Refine and suggests that effective long-horizon distillation requires teaching at the right depth, the right decision, and the right token.

Abstract

On-policy distillation (OPD) transfers the capabilities of a large language model to a smaller student by providing teacher supervision on the student's own rollouts. In long-horizon agentic tasks, however, uniform token-level matching can allocate supervision poorly: a large local discrepancy need not improve future behavior, while consequential guidance may be beyond the current student's reach or fail to persist without privileged input. We formulate long-horizon OPD as hierarchical supervision allocation and argue that productive guidance lies at the intersection of future utility and current learnability. Crucially, this intersection evolves as the student learns. Based on this principle, we propose LENS-OPD, a coarse-to-fine framework that organizes supervision through Locate, Validate, and Refine. Locate adapts trajectory exposure to the student's evolving competence and proposes a candidate decision for intervention. Validate tests whether teacher guidance at that decision improves the same student's subsequent behavior. Refine internalizes the beneficial guided behavior into the deployable policy and concentrates token-level supervision on decisive teacher-student conflicts within the validated turn. These stages are nested: each finer allocation is conditioned on the coarser decision, rather than being optimized as an independent importance score. Experiments across multiple long-horizon agent benchmarks and student-teacher configurations show that LENS-OPD consistently improves task performance over vanilla OPD and strong curriculum- and selection-based baselines. Our results suggest that effective long-horizon distillation requires teaching at the right depth, the right decision, and the right token.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Teacher Should Think Ahead: Adaptive Continuations for Reliable On-Policy Distillation

This work identifies Teacher Uncertainty Contraction (TUC), a systematic phenomenon whereby the teacher's predictive uncertainty decreases as it continues from a student-generated prefix, and proposes Adaptive-Continuations On-Policy Distillation (AC-OPD), which augments informative states along student rollouts with t...

Jin-Gang Zhou, Yu-Yi Zhou, Hai-Yang Guo et al. · 3 citations
#machine learning Preprint Oct 2026

DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation

DiffGate is introduced, an outcome-gated objective that combines GRPO with selective, bounded teacher guidance, and improves pass@8 across all four model--domain settings, demonstrating improved solution coverage under the evaluation protocol.

Karn Tiwari, V. Chordia, P. PrathoshA · 0 citations
#artificial intelligence Preprint Oct 2026

Stochastic Teacher Intervention for Agentic On-Policy Distillation

STI-OPD, a stochastic teacher intervention framework for multi-turn agentic OPD, and an Importance-Weighted Reverse KL objective that corrects the token sampling mismatch between teacher-generated responses and the student policy to preserve the original OPD objective are introduced.

Jun-Nan Liu, Lin-Hao Luo, Zhijun Chen et al. · 0 citations

Recursive Self-Improvement via On-Policy Distillation for Reasoning

DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks, and its comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks.

Shang-Jian Yin, Ze-Hao Zhao, Kavosh Asadi et al. · 6 citations
#artificial intelligence Preprint Oct 2026

A Good Self-Teacher Meets the Student Where They Are: Joint On-Policy Learning and Teaching

Reinforcement Learning (RL) from outcome rewards suffers from sparse supervision, particularly on difficult, long-horizon tasks where successful trajectories are rare and costly to generate. On-Policy Distillation (OPD) offers an attractive alternative by providing dense token-level supervision from a stronger teacher...

Randy Ardywibowo, Arnav Dalal, Jian-Tao Jiao · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.