SurveyUQBench: Recovering Complex-Survey Designs and Design-Based Uncertainty from Official Documentation

Abstract

Reliable public-survey analysis combines plausible coding and weighted point estimation with the recovery of the correct weights, strata, primary sampling units, replicate-weight convention, domain rule, degrees of freedom, and confidence-interval rule fro m official documentation. We introduce SurveyUQBench, a benchmark/resource that evaluates this complete path to design-based uncertainty. It contains 360 tasks drawn from 12 public survey programs and five variance mechanisms, together with official-document evidence qrels, evaluation-hidden design manifests, reference R analyses, and deterministic graders for evidence, design recovery, and estimate/standard-error/confidence-interval correctness. Reference analyses and clean replays cover the full resource, and mutation tests confirm that the deterministic grader distinguishes valid references from targeted mutations. Direct model responses achieve broad compliance with the response contract; strict NumericUQ and FullTask both record 0 under the joint criteria. On a fixed task-model subset, submission - native execution achieves 23 successful invocations; deterministic interface normalization increases this to 89, yielding 9 finite estimate/SE pairs and 2 NumericUQ passes —gold-input interventions further separate evidence, design, interface, and numerical stages. SurveyUQBench therefore provides a traceable evaluation resource, a reproducible baseline, and stage-specific signals for measuring future progress.