FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks

Abstract

A paraphrase-quality audit of MathCheck (ICLR 2025) detected 4 semantically incorrect paraphrases in 129 groups (3.1%); removing them drops GPT-4o from rank 2 to rank 4 and elevates Claude Haiku and DeepSeek V3 above it. Cross-model unanimity found these errors automatically for under $10; in our own dataset the same protocol found that 47% of auto-generated connective-variation paraphrases were semantically incorrect. That flaw compounds a deeper measurement gap. On the theorems every model answers correctly in canonical form, a semantically equivalent rewording still flips the answer for 5-42% of them: Claude Haiku 4.5 flips a known-correct answer about 9x as often as DeepSeek V3 (41.9% vs. 4.7% on the 43 theorems all models share), a difficulty-controlled gap we call the Conditional Inconsistency Rate (CIR). The CIR ranking replicates on an independent 103-theorem set. Raw Semantic Consistency Rates (SCR) span 50-82% across 9 models, but this spread is mostly a mechanical consequence of compounding accuracy over k ~ 6 paraphrases; what CIR isolates is the within-theorem clustering of failures. Because no model dominates on all paraphrase families, the families a benchmark includes implicitly fix its model ranking (a No-Free-Benchmark corollary). FormInv supplies the audit protocol, CIR and SCR as invariance measures, evaluated on 9 models across 366-811 items whose canonical statements are Lean4-verified, and FormInvSelector for regime-aware model selection.