Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards
It is shown that a widely used dataset's stated assumption of no intra-group annotator variability is false, and it is shown that an LLM judge tracks the crowd pool over the expert pool on all three models the authors test, including one from a different vendor.