Zhekun Shi

Statistical impossibility and possibility of aligning LLMs with human preferences: From Condorcet paradox to Nash equilibrium

It is proved that Condorcet cycles exist with probability converging to one exponentially fast under a general probabilistic preference model called the Luce model, thereby demonstrating the impossibility of fully aligning human preferences using reward-based approaches such as reinforcement learning from human feedbac...