How to post-train on a surrogate: Envelope sampling mitigates reward hacking
This work proposes envelope sampling, a theoretically-grounded method for judge recalibration that seeks to minimize an upper bound on the regret of the post-trained model under the assumption that the human reward and re-calibrated reward lie in an $L^2$ ball around the judge.
Sanjit Dandapanthula, Shuvom Sadhuka, Samir Khan et al.
· 0 citations