Yu-Run Yuan

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-policy sampling. We study when such off-policy deployment is justified, and what roles its core ingredients, incl...