A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards
This paper compares two low-cost choices retrospectively -- a cost-reducing choice and a balanced choice -- with estimated grading cost reductions and average post-training score gaps of 1-3 points from the best evaluated training verifier, and establishes that verifier choices are interchangeable.