Why Fisher's exact test is not exact: achieved size, wasted power, and a routing rule for 2×2 tables (m02d) -- Reproducibility bundle

Abstract

Why Fisher's exact test is not exact: achieved size, wasted power, and a routing rule for 2×2 tables William J. Dwyer, MD, MPH, FAAP — Department of Mathematics and Statistics, University of Massachusetts Lowell. ORCID 0009-0004-0855-7222. Concept DOI (always resolves to the latest version): 10.5281/zenodo.22238930. This release (v1.0.5):10.5281/zenodo.22238931. What this is The reproducibility deposit for the m02d paper. "Fisher's exact test" is exactly valid — its true rejection rate never exceeds the nominal α — but it is not size-exact: because the 2×2 reference distribution is discrete, no table boundary falls at α, so the achieved (unconditional) size sits below nominal. At small designs the test spends only 26–66% of its α budget, and wasted size is wasted power. The paper computes the achieved size, the wasted-power cost, and the significance-flip behaviour of Fisher's exact against a conservative→liberal roster (Yates χ², Fisher, mid-p, Boschloo, Barnard, Pearson χ², Cressie–Read λ=2/3, likelihood-ratio G), and distills the result into a routing rule for the 2×2. Everything is deterministic exhaustive enumeration — two independent binomials, no Monte Carlo, no random seed — so there is no simulation error: every number, table, and figure reproduces exactly from a named script. The keynote A real-data scan of 6,030 public 2×2 tables (757 pydataset/Rdatasets datasets) shows the tests agree essentially everywhere away from the threshold but disagree in ~84% of cases inside the Fisher-p band 0.05–0.07 — that is, the tests are miscalibrated relative to one another exactly where we use them to draw the significant/not-significant line. Flips are read on a three-tier taxonomy relative to Boschloo (the valid frontier): conservative below it, valid on it, liberal above it. What the deposit contains Manuscript (author + anonymized, .md), the novelty / prior-art companion, and the cover letter (The American Statistician). Reproducibility apparatus (rerun/) — the exact-enumeration engine achieved_size.py (tie-aware conditional p-values, validated against SciPy to 2.2e-16) and the deterministic sweep drivers: power_and_flips.py, size_skew_sweep.py, flip_boundary_sweep.py, roster_sweep.py (the full comparator roster, including the Cressie–Read λ=2/3 power-divergence member), and public_flip_scan.py (the real-corpus scan). Each writes its locked JSON/CSV. Figures (figures/) — the achieved-size and wasted-power panels, the size×skew heat maps, the α=0.05 flip-boundary maps, the χ²-inclusive roster figures, the real-data flip-incidence figure, and the routing flowchart (Figure 7). Interactive demonstrator honest_fisher.html — a self-contained in-browser calculator with the full comparator taxonomy and the live-highlight routing flowchart; its JavaScript engine reproduces the deposited numbers and self-checks a parity badge on load. Deep-dive record (deepdives/) — the supporting analyses, including the size/skew, flip-boundary, χ²-roster, real-data, and computed-clearance write-ups, and the M0u-bundle assessment that reconciles m02d with the author's earlier Monte-Carlo program (below). LICENSE, CITATION.cff, MANIFEST.txt (SHA-256 of every file). The Cressie–Read λ=2/3 reconciliation The roster now carries Cressie–Read λ=2/3, the best-calibrated member of the power-divergence family that also contains Pearson (λ=1) and the G-test (λ=0). Across R×C shapes it is the best-calibrated test in the author's companion Monte-Carlo study; on the pure 2×2 that is m02d's subject, its exact achieved size is 0.062 / 0.053 / 0.055 / 0.055 at the representative designs — still mildly liberal, because small-sample discreteness bites it too. So the paper names λ=2/3 as the calibrated member of the χ² family, scopes the "common χ² defaults overshoot" claim to Pearson/Yates/G, and shows that even the well-chosen asymptotic statistic does not escape the 2×2 discreteness the exact unconditional tests handle — turning a potential referee objection into a strengthening of the routing rule. The companion 18-test Monte-Carlo study independently confirms m02d's Boschloo/Barnard power gap over Fisher (+3 to +10 pp by simulation vs+3.7 to +11.1 pp by exact enumeration here). All computation is deterministic exact enumeration; code is released under the MIT License, and text, figures, and data under CC BY 4.0. The public 2×2 corpus is built from the Rdatasets collection via pydataset; no data are redistributed beyond the illustrative tables cited in the paper. How to cite Please cite this deposit if you use the package or the method. Citing the concept DOI references the work in general and always resolves to the latest version; cite a specific version DOI to point at an exact snapshot. Dwyer, W. J. (2026). Why Fisher's exact test is not exact: achieved size, wasted power, and a routing rule for 2×2 tables — reproducibility deposit [Software]. Zenodo. https://doi.org/10.5281/zenodo.22238930 BibTeX: bibtex @software{dwyer_m02d_2026, author = {Dwyer, William J.}, title = {Why Fisher's exact test is not exact: achieved size, wasted power, and a routing rule for 2x2 tables --- reproducibility deposit}, year = {2026}, publisher = {Zenodo}, doi = {10.5281/zenodo.22238930}, url = {https://doi.org/10.5281/zenodo.22238930}, orcid = {0009-0004-0855-7222} } The DOI above is the concept DOI (resolves to the latest version); to cite a specific release use that version's DOI in its place. When the accompanying journal article appears, please cite it as the primary reference for the method and this deposit as the reproducibility archive. Version history v1.0.6 — staged, not yet uploaded (m02d_fishers_exact_not_exact_reproducibility_v1.0.6.zip, md5 fc3d9dcd5574606e97e5e7f20b434024, 1,848,064 B, 58 files): prior-art citation-completeness pass. Added eight Crossref-verified references — Chen (2011), Neuhäuser & Ruxton (2025), Rudas (1986), Upton (1982), Little (1989), Berger (1996), Routledge (1994), and Hirji (2006) — and, in §8, framed the real-corpus keynote as the empirical, at-scale demonstration of the test-disagreement result Chen (2011) characterized theoretically; §6 now attributes the λ=2/3 calibration to Rudas (1986). No number, figure, table, or claim changed. Upload this version to mint its DOI. v1.0.5 — ✅ 10.5281/zenodo.22238931 (2026-09-01) (m02d_fishers_exact_not_exact_reproducibility_v1.0.5.zip, md5 31f2a7bfeda570ed402dd1aa3fb619e6, 1,846,004 B, 58 files): added the Cressie–Read λ=2/3 member to the roster engine (exact 2×2 achieved size 0.062/0.053/0.055/0.055) and its column to Table 1; scoped the χ²-overshoot claim to the common defaults and named λ=2/3 as the calibrated member (§6); added the Cressie & Read (1984) / Read & Cressie (1988) references and the companion Monte-Carlo self-citation (§9); folded in the M0u-bundle vs m02d assessment deep-dive. Numbers re-verified against the deposited JSON; deterministic (identical md5 on two runs). v1.0.4 — md5 b59f667573823962f999ff040ec591e4, 1,837,352 B, 57 files: routing-flowchart computed-clearance finalizer (Figure 7 and the demonstrator SVG from one audited spec). v1.0.3 — md5 0f01c96e0eb836b2572109446db1b20d, 1,833,339 B: manuscript body expanded to fold the figures/tables with organized in-text callouts; anon regenerated; audits PASS. v1.0.2 — md5 29caf50da2d7bec2409f42098d979e64, 1,830,784 B: full comparator roster and the real-data flip scan added; demonstrator enriched with the taxonomy. v1.0.1 — md5 e0733897cad80d8178d17c8ffcb6943c, 1,714,057 B: size×skew and flip-boundary sweeps + figures. v1.0.0 — md5 54286e0b6841778c71cbea2b20a56897, 1,703,321 B, 48 files: first build — the exact achieved-size engine, the power/flip computation, and the honest_fisher.html demonstrator. Provenance: every number traces to a named, seed-free exact-enumeration script and is validated against SciPy; the demonstrator reproduces the deposited Python. Related work: the companion 18-test Monte-Carlo contingency-table program (M0/M0u), folded into the repo as m0u_work and cited for the Cressie–Read result, the Rényi negative result, and the independent power-gap confirmation.