The Flaw of Averages: Measuring Benchmark-Level Distributional Robustness
The findings show that aggregate scores can misrepresent broad competence when performance is unevenly distributed, and recommend reporting benchmark Harmony alongside aggregate accuracy as a diagnostic of benchmark representativeness when interpreting claims about broad model competence.
Arda Uzunoglu, Tian-Jian Li, Daniel Khashabi
· 0 citations