LLM Value Functions for Credit Assignment in Agentic RL
LLM Value Function RL is introduced, which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic, and consistently and substantially outperforms both GRPO and PPO.