Dhruv Puri

Compression Precedes, Content Decides: Emergent Misalignment in Reward Models

It is shown that narrow fine-tuning reward models to prefer harm in one domain (medical or financial) generalizes to seven held out domains and that reward models are more vulnerable to preference inversions compared to their language model counterparts.