Prevalence Shift and ICU Mortality Miscalibration Across Institutions
Abstract
Machine learning models for predicting in-hospital mortality in intensive care units (ICUs) are increasingly deployed across hospital systems with heterogeneous patient populations and outcome rates. Despite growing awareness of dataset shift, the calibration consequences of hospital-level prevalence variation remain uncharacterized at scale. This paper investigates hospital-level outcome prevalence as a primary mechanistic driver of calibration failure under cross-institutional deployment. Four classifiers-Logistic Regression, Random Forest, XGBoost, and LightGBM-were trained on MIMIC-IV $(N=45,587$ patients, $\pi_{0}=0.108)$ and evaluated across 185 hospitals in the eICU Collaborative Research Database $(N=159,224$ patients). Hospital-level prevalence difference explained substantial variance in Expected Calibration Error (ECE: $r=0.52-0.75, p<10^{-14})$, while the area under the receiver operating characteristic curve (AUROC) showed no significant correlation $(r=-0.03-0.22, p=0.01-0.75)$. A closed-form Bayesian prevalence correction requiring only the local mortality rate, and no labeled target data, reduced mean ECE from 0.071-0.085 to 0.027-0.037, comparable to Platt scaling applied with $\mathbf{3 0 \%}$ labeled data $(\mathbf{E C E}=0.022-0.024)$. Synthetic validation with controlled prevalence shifts confirmed causality, and empirical attribution analysis revealed that prevalence shift explains approximately 55-65% of calibration error for well-calibrated models. While the Bayesian correction formula is grounded in established labelshift theory [10]–[12], the novel contributions of this work are: (a) hospital-level empirical quantification of ECE degradation across 185 real institutions, (b) demonstration of the AUROC/ECE dissociation at this scale, (c) causal validation via controlled synthetic prevalence shift, and (d) an attribution decomposition quantifying prevalence's share of calibration error. These results establish prevalence mismatch as a dominant, analytically tractable source of deployment miscalibration, correctable using a single population statistic available at every institution.