Skip to content
Preprint

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

Oct 2026 · 0 citations · 28 references
Computer Science

TL;DR

This work reuses historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training and analyzes when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student.

Abstract

Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$-$5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$-$90.1\%$, while maintaining comparable multi-domain performance.

View source

Similar papers

REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse

On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its reliance on frequently refreshed student rollouts often incurs substantial generation cost. We introduce REVO, an off-policy distillation framework that improves rollout effici...

Yu-Xiao Yang, Shang-Zhe Li, Tian-Run Yu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Rethinking Self-Distillation for Multi-Teacher Capability Merging

Combining capabilities of multiple expert models trained starting from the same base checkpoint has become increasingly common in frontier language-model post-training. Recent trends suggest that multi-teacher on-policy distillation (MOPD) outperforms conventional off-policy methods. However, despite the higher inferen...

Roy Xie, Dan Friedman, Feng Nan et al. · 1 citation
#artificial intelligence Preprint Oct 2026

SAPD: Learning from Agent Experience without Environment Interaction

Step-Advantage Policy Distillation (SAPD) is proposed, a method for learning from offline trajectories using relative step advantages, allowing the student to generate responses conditioned on recorded histories, and introduces relative step advantage signals to stabilize training.

Jun-Cheng Diao, Hai-Bin Wang, Wei Shi et al. · 0 citations

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to re...

Seong-Jang Kim, Chaeyun Jang, Noah Lee et al. · 2 citations · ⚡1
#machine learning Preprint Oct 2026

RM-OPD: On-Policy Distillation for Generative Reward Models

Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly o...

Qing-Lin Su, Li-Cheng Pan, Zhen Fang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.