MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
Most medical AI benchmarks measure whether a model knows the correct answer. MedFailBench asks a different question: which safety boundary failed? We present a synthetic benchmark and failure atlas built by a clinician. The resource labels medical AI errors by severity from 1 to 5 and safety gate type: missed urgent escalation, unsafe remote dosing, unsafe discharge reassurance, evidence fabrication, unsafe protocol execution, and source support gap. The current public release (v0.2.1) contains 44 synthetic cases reviewed by a clinician, with severity annotations, a public Hugging Face Space source, a safety gate taxonomy, a clinical severity rubric, and an automated pipeline for archiving model response screening runs. Forty cases have a populated safety gate field, and four require gate completion. No patient data, clinical validation claims, or model rankings are included. MedFailBench is released under Apache-2.0 and CC-BY-4.0 and carries the Zenodo DOI 10.5281/zenodo.21205535.