Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2x2 tables -- m02d Reproducibility deposit

Abstract

Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2×2 tables William J. Dwyer, MD, MPH, FAAP — Department of Mathematics and Statistics, University of Massachusetts Lowell. ORCID 0009-0004-0855-7222. Concept DOI (always resolves to the latest version): 10.5281/zenodo.22238930. This release (v1.1.14):10.5281/zenodo.22262453, published Sep 2, 2026; the concept DOI above resolves to it. The previous release, v1.1.13, is 10.5281/zenodo.22262451; earlier, v1.1.5 is 10.5281/zenodo.22260812 and v1.1.4 is 10.5281/zenodo.22260570. What this is The reproducibility deposit for the m02d paper. "Fisher's exact test" is exactly valid — its true rejection rate never exceeds the nominal α — but it is not size-exact: because the 2×2 reference distribution is discrete, no table boundary falls at α, so the achieved (unconditional) size sits below nominal. At small designs the test spends only 26–66% of its α budget, and wasted size is wasted power. The paper computes the achieved size, the wasted-power cost, and the significance-flip behaviour of Fisher's exact against a conservative→liberal roster (Yates χ², Fisher, mid-p, Boschloo, Barnard, Pearson χ², Cressie–Read λ=2/3, likelihood-ratio G), and distills the result into a routing rule for the 2×2. Everything is deterministic exhaustive enumeration — two independent binomials, no Monte Carlo, no random seed — so there is no simulation error: every number, table, and figure reproduces exactly from a named script. The keynote A real-data scan of 6,030 public 2×2 tables (757 pydataset/Rdatasets datasets) shows the tests agree essentially everywhere away from the threshold but disagreement spikes on each decision line — ~67% of tables within ±0.02 of 0.05 and ~76% within ±0.02 of the 0.10 trend line get a test-dependent verdict — so the tests are miscalibrated relative to one another exactly where we use them to draw the significant/not-significant line, and the trend line inherits the same spike. Flips are read on a three-tier taxonomy relative to Boschloo (the valid frontier): conservative below it, valid on it, liberal above it. What the deposit contains Manuscript (author + anonymized, .md), the Derivations companion (D1–D6: conditional validity and the achieved-size shortfall; mid-p as the mean of the median-unbiased rule with E₀[mid-p] = ½; Boschloo's uniform power dominance; the nesting that makes flips one-directional; the atom-lattice gap and exact Boschloo region behind Figure 3; and why conditioning forfeits the budget), the novelty / prior-art companion, and the cover letter(The American Statistician). Reproducibility apparatus (rerun/) — the exact-enumeration engine achieved_size.py (tie-aware conditional p-values, validated against SciPy to 2.2e-16) and the deterministic sweep drivers: power_and_flips.py, size_skew_sweep.py, flip_boundary_sweep.py, roster_sweep.py (the full comparator roster, including the Cressie–Read λ=2/3 power-divergence member), and public_flip_scan.py (the real-corpus scan). Each writes its locked JSON/CSV. Figures (figures/) — the achieved-size and wasted-power panels, the size×skew heat maps, the α=0.05 flip-boundary maps, the χ²-inclusive roster figures, the real-data flip-incidence figure, the routing flowchart (now Figure 9), and the per-test direction butterfly (Figure 8). Interactive demonstrator honest_fisher.html — a self-contained in-browser calculator with the full comparator taxonomy and the live-highlight routing flowchart; its JavaScript engine reproduces the deposited numbers and self-checks a parity badge on load. Deep-dive record (deepdives/) — the supporting analyses, including the size/skew, flip-boundary, χ²-roster, real-data, and computed-clearance write-ups, and the M0u-bundle assessment that reconciles m02d with the author's earlier Monte-Carlo program (below). LICENSE, CITATION.cff, MANIFEST.txt (SHA-256 of every file). The Cressie–Read λ=2/3 reconciliation The roster now carries Cressie–Read λ=2/3, the best-calibrated member of the power-divergence family that also contains Pearson (λ=1) and the G-test (λ=0). Across R×C shapes it is the best-calibrated test in the author's companion Monte-Carlo study; on the pure 2×2 that is m02d's subject, its exact achieved size is 0.062 / 0.053 / 0.055 / 0.055 at the representative designs — still mildly liberal, because small-sample discreteness bites it too. So the paper names λ=2/3 as the calibrated member of the χ² family, scopes the "common χ² defaults overshoot" claim to Pearson/Yates/G, and shows that even the well-chosen asymptotic statistic does not escape the 2×2 discreteness the exact unconditional tests handle — turning a potential referee objection into a strengthening of the routing rule. The companion 18-test Monte-Carlo study independently confirms m02d's Boschloo/Barnard power gap over Fisher (+3 to +10 pp by simulation vs +3.7 to +11.1 pp by exact enumeration here). All computation is deterministic exact enumeration; code is released under the MIT License, and text, figures, and data under CC BY 4.0. The public 2×2 corpus is built from the Rdatasets collection via pydataset; no data are redistributed beyond the illustrative tables cited in the paper. How to cite Please cite this deposit if you use the package or the method. Citing the concept DOI references the work in general and always resolves to the latest version; cite a specific version DOI to point at an exact snapshot. Dwyer, W. J. (2026). Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2×2 tables — reproducibility deposit [Software]. Zenodo.https://doi.org/10.5281/zenodo.22238930 BibTeX: bibtex @software{dwyer_m02d_2026, author = {Dwyer, William J.}, title = {Why Fisher's exact test is least exact where it matters most: achieved size, wasted power, and a routing rule for 2x2 tables --- reproducibility deposit}, year = {2026}, publisher = {Zenodo}, doi = {10.5281/zenodo.22238930}, url = {https://doi.org/10.5281/zenodo.22238930}, orcid = {0009-0004-0855-7222} } The DOI above is the concept DOI (resolves to the latest version); to cite a specific release use that version's DOI in its place. When the accompanying journal article appears, please cite it as the primary reference for the method and this deposit as the reproducibility archive. Version history v1.1.14 — ✅ 10.5281/zenodo.22262453 (published 2026-09-02) (m02d_fishers_exact_not_exact_reproducibility_v1.1.14.zip, md5 4977e10c1b632054e3e6609176dde2fd, 3,722,698 B, 94 files): the Boschloo-dominates-Fisher machinery, made explicit. Section 6 now shows the one-line power-dominance mechanism rather than only asserting it: Fisher's own conditional test already has unconditional size ≤ α, so the calibrated Boschloo threshold c satisfies c ≥ α, and because both tests threshold the same statistic the rejection set {Fisher p ≤ α} sits inside {Fisher p ≤ c} table for table — Boschloo rejects wherever Fisher does and more, so it is uniformly at least as powerful, a guarantee that would fail for a differently-ordered unconditional statistic. Derivation D3.4 gains the explicit note that Boschloo and the score-based Barnard test are mutually non-nested (neither uniformly dominates; Barnard's occasional power edge is budget spent, not dominance), and the SN supplement gains a "Why Boschloo, not Barnard, is the valid frontier" note beside the comprehensive roster. Exposition over the existing exact-enumeration output; no computed result, figure, or table changed. Deterministic (identical md5 on two runs). Supersedes v1.1.13. v1.1.13 — ✅ 10.5281/zenodo.22262451 (published 2026-09-02) (m02d_fishers_exact_not_exact_reproducibility_v1.1.13.zip, md5 371a192f5a3ea8ad1e552043a3d68562, 3,721,908 B, 94 files): the comprehensive-roster and full-budget-grid build-out, and the worked-example correction. Adds the sixteen-statistic exact achieved-size-and-power engine comprehensive_roster.py, the full per-test α-budget-grid enginebudget_grid_all.py, the vectorized Boschloo helper boschloo_fast.py (an O(P·K) replacement for the O(n⁴) per-cell sup, verified cell-for-cell against the naive loop, which makes exact enumeration to n = 100 tractable), and the supplement table generator splice_supplement_tables.py. The SN supplement's per-test α-budget grids now run n = 6, 8, …, 20, 25, …, 65, 80, 100 and the big-tent size/power tables use the columns n = 8, 10, 16, 25, 40, 65, 100. Also corrects the worked-example Table S1 Cressie–Read λ = 2/3 value — the deposited 0.1881 ("not significant") was wrong; the exact statistic gives p = 0.0287 (significant), cross-checked against scipy.stats.power_divergence — expands that table to the full sixteen-test roster, and rewrites the accompanying narrative, which had leaned on the wrong number. Rolls up the staged v1.1.6–v1.1.12 (the SN-supplement rework: per-test α-budget grid atlas, the diverging blue–white–red under/over color standard, the neutral-hatch Figure 8, and the m0-sourced comparator roster). Deterministic (identical md5 on two runs). Supersedes v1.1.5. v1.1.5 — ✅ 10.5281/zenodo.22260812 (published 2026-09-02) (m02d_fishers_exact_not_exact_reproducibility_v1.1.5.zip, md5 d96e845d54d12ae496fe78d7440ca4c7, 3,413,155 B, 79 files): availability statement consolidated into Declarations; placement made a canonical build check. The data-availability text was appearing twice — as the Availability of data and materials. statement inside Declarations and as a standalone ## Data and code availabilitysection bolted on after the References, which had drifted (the Declarations copy still read placeholder wording while the live DOI lived only in the bolted-on section). Consolidated into the single Declarations statement (scope + no-redistribution; the reproducibility package on Zenodo under the concept DOI as a live link