Quantifying the Accuracy-Explainability Trade-off in Statistical and Machine Learning Models Across Multiple Clinical Datasets: A Cross-Pathology Validation Framework

Abstract

Objective: The rapid digitisation of healthcare has necessitated the development of high-performance predictive models for clinical diagnostics. While traditional statistical frameworks such as Multiple Logistic Regression (MLR) are favoured for their intrinsic interpretability, they often encounter an accuracy ceiling when applied to complex, non-linear clinical data. Conversely, advanced machine learning ensembles such as eXtreme Gradient Boosting (XGBoost) offer superior classification ability but suffer from the black-box problem. This study aims to quantify the accuracy-explainability trade-off by comparing a transparent baseline against an Explainable AI (XAI)-augmented pipeline across three distinct clinical datasets. Methods: This study employs a multi-cohort comparative framework using the Heart Failure Clinical Records (n = 299), Wisconsin Breast Cancer Diagnostic (n = 569), and Pima Indians Diabetes (n = 768) datasets, representing cardiovascular, oncological, and metabolic diagnostic spaces, respectively. The methodology incorporates a Phase 2 Clinical Assay Development framework with 10-fold stratified cross-validation and Parallelised Leave-One-Out Cross-Validation (LOOCV) as a high-fidelity sensitivity audit. DeLong’s test for correlated ROC curves and the Ramsey RESET test were employed to evaluate the significance of performance and structural misspecification. A 2,000-iteration bootstrap stability audit was executed to verify estimator consistency. Shapley Additive exPlanations (SHAP) were applied as a post-hoc audit framework, with reliability verified via an Explanatory Stability Score (S). Results: Model superiority is dictated by a complexity threshold tied to data geometry. In high-dimensional oncological data, XGBoost demonstrated significant superiority over MLR (AUC 0.994 vs. 0.941; p < 0.001), structurally mandated by a massive RESET misspecification (p < 10−52). A LOOCV sensitivity check revealed a statistical tie (p = 0.676), with a bootstrap 95% CI of [−0.0047, 0.0066] that contains zero, positioning MLR as a robust and parsimonious champion when target leakage is controlled. The cardiovascular cohort exhibited a stable tie (95% CI: [−0.0564, 0.0378]), and the metabolic cohort demonstrated clear linear sufficiency (RESET p = 0.112), resulting in significant MLR superiority over XGBoost (p = 0.0006). SHAP stability metrics (S > 0.80) confirmed that audited signatures represent robust biological markers rather than stochastic artefacts. Conclusion: This research establishes that the accuracy-explainability trade-off is not a static penalty but a dynamic threshold dictated by clinical dimensionality and data geometry. By providing a mathematically validated framework incorporating structural diagnostics, bootstrap stability, and SHAP-based explainability, this study demonstrates that diagnostic tools can achieve peak accuracy without sacrificing the transparency required for clinical accountability.