MESA: Requirements-Oriented Review of AI Benchmarks as Measurement Instruments

Abstract

Requirements Engineering (RE) for AI increasingly relies on benchmark scores to justify model selection, safety claims, and procurement decisions. Yet AI benchmarks are rarely reviewed as measurement instruments: their intended constructs, scoring assumptions, evidence limits, and validity arguments are often left implicit. This paper introduces the Meta-Evaluation Standard for AI Benchmarks (MESA), a requirements-oriented framework for auditing AI benchmarks before their scores are used to support capability claims. MESA adapts the European Federation of Psychologists' Associations (EFPA) test review model to the field of AI benchmarks, and is divided into two parts. Part A is descriptive: it records the benchmark's provenance, task design, target systems, scoring procedure, reporting outputs, and lifecycle materials. Part B is evaluative: it assesses whether the available evidence supports the intended interpretation of benchmark scores, including reliability, validity, contamination risk, comparability, maintenance, and claim proportionality. We demonstrate MESA on three contrasting benchmarks: Humanity's Last Exam (HLE), ARC-AGI-2, and DesignQA. The reviews show that HLE is best interpreted as a difficult closed-ended academic question-answering benchmark, not as a standalone basis for high-stakes ranking or broad intelligence claims. ARC-AGI-2 is best interpreted as an exact-grid abstraction benchmark under specified evaluation conditions, not as a standalone test of general intelligence. DesignQA is best interpreted as a documentgrounded engineering-requirements question-answering benchmark, with limited claims but important gaps in uncertainty reporting, licensing, human baselines, and test-set governance. MESA provides an open-source and replicable quality standard for specifying, auditing, and validating benchmark-based claims in AI RE.