Beyond the black box in healthcare operations: a structured literature review and governance framework for explainable AI in clinical process intelligence systems

Abstract

Healthcare artificial intelligence has produced substantial diagnostic capabilities, yet clinical operations remain underserved by AI transparency research. Existing reviews of Explainable AI (XAI) in healthcare concentrate on diagnostic imaging, oncology, radiology, and clinical decision support. The governance needs of clinical process intelligence (CPI) systems, which support patient flow, resource allocation, throughput management, and care coordination, remain comparatively underdeveloped. This original research article reports a structured literature review with systematic search and screening, followed by framework development. It characterizes healthcare XAI research across application domains and introduces the CPI-XAI Governance Framework, a theory-informed model for operationalizing algorithmic transparency as an organizational information-systems capability in clinical process intelligence environments. We searched PubMed/MEDLINE, Scopus, Web of Science, IEEE Xplore, ACM Digital Library, and CINAHL for peer-reviewed studies published from January 2015 to April 2026. Screening and extraction used transparent procedures adapted from PRISMA flow-reporting practices. Two authors independently screened records and extracted domain, XAI method, setting, evaluation, and governance variables; disagreements were resolved by author adjudication. Framework development used a documented hybrid deductive-inductive coding pipeline, source-contribution rules, negative-case review, and an auditable evidence-to-framework mapping anchored in IT Capability Theory, the DeLone–McLean IS Success Model, implementation science, sociotechnical systems theory, and health AI governance guidance. Of 187 included studies, 175 (93.6%) addressed diagnostic, disease-specific, imaging, or general clinical decision-support contexts, while operational and workflow-embedded AI accounted for 9 studies (4.8%). The primary result is descriptive: XAI research in healthcare remains concentrated in diagnostic domains, while explicit explainability-governance work for clinical operations is sparse within the search boundary. Five evidence gaps informed the framework: limited operational XAI evaluation standards, role-blind explanation design, weak causal and counterfactual support, poor workflow embedding, and absence of longitudinal XAI governance. A preliminary 0–15 scoring rubric and two hypothetical organizational assessments demonstrate how the framework can be applied analytically; they do not constitute external validation. Healthcare XAI remains concentrated in diagnostic and clinical decision-support contexts, while explicit explainability-governance research for operational AI systems remains thinly represented. The CPI-XAI Governance Framework addresses this gap by treating explainability as an organizational capability rather than a model-level technical feature. The framework is preliminary and theory-informed; it should be empirically validated before use as a measurement instrument.