Trust the Critic More
Standard language model RL algorithms credit every token of a long rollout with the same advantage determined by the terminal reward. Actor-critic methods can provide finer-grained credit assignment, but learned critics are generally considered too inaccurate to trust when training LLMs with RL. In recent works, even w...
Kai-Yue Wen, Luke Bailey, Arvind V. Mahankali et al.
· 0 citations