Temporal-difference (TD) learning trains neural representations using bootstrapped targets generated by evolving value estimates, yet how predictive features emerge and propagate under incomplete neural updates remains poorly understood. We investigate this problem using finite-width two-layer ReLU critics on a constru...
Shang-Zhe Li, De-Chen Zhang, Di-Fan Zou et al.· 0 citations
On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its reliance on frequently refreshed student rollouts often incurs substantial generation cost. We introduce REVO, an off-policy distillation framework that improves rollout effici...
Dually Regularized AIL is the first algorithm to simultaneously achieve $\widetilde{O}\left(\frac{1}{\epsilon}\right)$ sample complexity in both expert demonstrations and online interactions for this regularized AIL objective, even with stochastic experts.
Hang Zhou, Shang-Zhe Li, А. А. Браверман et al.· 0 citations
An RL perspective on OPD is provided that offers both a principled interpretation and a practical route toward more effective and rollout-efficient language model distillation.
Shang-Zhe Li, Yu-Xiao Yang, Tian-Run Yu et al.· 0 citations
CIS replaces the unbounded second moment that governs the error of exact importance sampling with a term bounded by a constant, at the cost of a bias controlled by the truncated excess, in evaluation across three mixture-of-experts models and five mathematical reasoning benchmarks.
Tian-Run Yu, Kai-Xiang Zhao, Shang-Zhe Li et al.· arXiv.org· 1 citation
We study length inflation in on-policy distillation (OPD), where student responses can become excessively long and even exhaust the generation budget. We identify \emph{termination-token mismatch} between base students and post-trained teachers as an important source of this behavior. Across Qwen3, Llama, and Gemma, th...
Yu-Xiao Yang, Tian-Run Yu, Shang-Zhe Li et al.· 4 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.