How Many Humans Are 32 LLM Judges Worth?
A panel's human-equivalent size is target-specific. Matching a fixed 32-judge panel to empirical human label distributions on three ChaosNLI tasks yields two distinct effective sizes: distributional-error matching gives $\nu_{\mathrm{MSE}}=2.304$, $3.750$, and $3.445$, whereas spectral matching gives $\nu_H=4.242$, $6....
Chao Li, Ying-Ying Yu, Yun-Feng Li
· 1 citation