Critic-Free Pretraining is introduced: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates.
Abstract
Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.
Offline reinforcement learning aims to learn effective policies from fixed datasets without online interaction, necessitating conservative constraints to mitigate the out-ofdistribution issue. Although existing approaches alleviate this issue through conservative constraints or policy regularization, they still struggl...
Hui-Zhi Wang, Yan Kong· International journal of sof...· 0 citations
This work revisits the conventional offline RL paradigm and proposes decoupling policy improvement from actor training, and trains the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic.
Xu-Yao Lin, Yixiang Shan, Jin-Ru Duan et al.· 2 citations
A Q-guided conditional variational autoencoder (Q-CVAE) is introduced that learns to generate expert policy priors directly from the online replay buffer, enabling the generation of high-value actions for guiding policy updates without relying on precollected expert trajectories.
Gong Gao, Weidong Zhao, Xianhui Liu et al.· IEEE Transactions on Neural...· 0 citations
This paper systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy, and proposes Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in onlin...
Perry Dong, Ronnie Polonsky, Dorsa Sadigh et al.· arXiv.org· 3 citations
Offline reinforcement learning (RL) must reconcile two competing requirements: policy updates should stay near dataset-supported actions to keep value estimates reliable, yet meaningful gains often require moving beyond the behavior distribution. We develop a geometric view of offline actor updates by modeling policies...
Online Self-Weighted Fine-Tuning is proposed, a simple method that augments SFT with online, trajectory-level weighting and offers a favorable compute-performance trade-off as a practical approach for fine-tuning small-to-medium LLMs on binary-verifiable reasoning tasks with only 2 online rollouts.
Hai-Quan Wen, Yi-Wei He, Bei Peng et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
