Identifying Key Explainability Components in Black-Box AI Models for Emergency Department Disposition Prediction
Abstract
Predicting patient disposition outcomes in the Emergency Department (ED)—specifically distinguishing between hospital admission and discharge home—is important for optimizing hospital capacity, staffing, and patient flow. While black-box Artificial Intelligence (AI) models such as Long Short-Term Memory (LSTM), K-Nearest Neighbors (KNN), Random Forest (RF), and Gradient Boosting Machines (GBM) achieve high predictive performance, their decision-making processes limit clinical interpretability and trust. This paper aims to evaluate and identify the key components of explainability across these four black-box models for ED disposition prediction. Using an analytical sample of 2,000 ED stays from MIMIC-IV-ED triage and demographic variables, we evaluated model performance across five standard classification metrics (Accuracy, Precision, Recall, F1-Score, and AUC) under an 80:20 stratified train-test split. Post-hoc feature attribution techniques—SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME)—were applied to identify dominant decision drivers and define core explainability dimensions. Random Forest achieved the highest predictive performance (Accuracy: 0.970, Precision: 0.963, Recall: 0.997, F1-Score: 0.979, AUC: 0.988), followed by KNN (Accuracy: 0.948, AUC: 0.977) and Gradient Boosting (Accuracy: 0.902, AUC: 0.944), whereas LSTM trained on single triage snapshots yielded lower performance (Accuracy: 0.875, AUC: 0.921). SHAP and LIME attributions highlighted clinician-assigned triage acuity (mean |SHAP| = 0.096), systolic blood pressure (0.065), body temperature (0.051), walk-in arrival mode (0.044), and oxygen saturation (0.042) as dominant decision factors. Synthesizing technical attributions with clinical requirements establishes five core explainability dimensions: Transparency, Interpretability, Fidelity, Consistency, and Actionability. Demonstrating predictive accuracy alone is insufficient for clinical adoption; establishing these grounded components provides a necessary foundation for human-centered decision support in emergency care.