Data Access and Quality Barriers in Large-Scale Administrative Health Data: A Reproducible, Information-Loss-Aware Harmonization Framework
TL;DR
The results show that, for heterogeneous administrative data, the key value lies not in scale alone but in the auditability of transformations, explicit loss documentation, and reproducibility of the harmonization process.
Abstract
Large-scale administrative hospital discharge data is a key resource for secondary health systems research, yet reuse is constrained by barriers of access, quality, interoperability, and semantic comparability. This paper presents and validates a reproducible, declarative, loss-aware harmonization framework for public discharge data that avoids full migration to a comprehensive common data model. The framework comprises a lightweight 14-field canonical model, versioned JSON crosswalks, a shared execution engine, a resilient file reader, schema validation, idempotency tests, value-domain checks, and an information-loss map. It was evaluated on public record-level discharge data from five jurisdictions on three continents (Korea, Brazil, Mexico, Chile, and New York State), comprising 561,966,231 harmonized records from 2001 to 2025. Validation demonstrated conformance to the declared source profiles for 82 of 99 files and full canonical conformance for 39, idempotency across all 99 files, 99.99% conformance with permitted value domains under an explicitly stated aggregation, and detection of source-level defects such as truncated files, malformed rows, and completeness anomalies. A marker-condition query for ischemic stroke (ICD-10 I63) showed that a single case definition executes consistently on the four sources retaining raw ICD-10 codes. The results show that, for heterogeneous administrative data, the key value lies not in scale alone but in the auditability of transformations, explicit loss documentation, and reproducibility of the harmonization process.