This systematic review provides a comprehensive analysis of XAI methods specifically applied to tabular healthcare data for classification tasks, revealing that SHAP remains the dominant post-hoc method, achieving strong model fidelity but showing inconsistent alignment with clinical expert reasoning.
Abstract
Explainable Artificial Intelligence (XAI) has emerged as a critical enabler for the adoption of machine learning models in high-stakes domains such as healthcare. While significant progress has been made in XAI for computer vision and natural language processing, tabular data—the predominant format of electronic health records—presents unique challenges and opportunities. This systematic review provides a comprehensive analysis of XAI methods specifically applied to tabular healthcare data for classification tasks. We examine 21 primary studies published between 2020 and the first half of 2026, covering three complementary perspectives: (1) intrinsically interpretable models, (2) post-hoc methods including LIME, SHAP, and their variants, and (3) evaluation frameworks that assess both model-centered fidelity and human-centered clinical alignment. Our analysis reveals that SHAP remains the dominant post-hoc method, achieving strong model fidelity but showing inconsistent alignment with clinical expert reasoning. Key findings include the significant impact of class imbalance on explanation consistency, the importance of clinician-centered evaluation, and the emergence of hybrid approaches integrating XAI with generative AI and transfer learning. We identify critical gaps, including limited adoption of XAI in AutoML pipelines, lack of standardized evaluation metrics, and predominance of single-institution validation studies.
Multimodal Artificial Intelligence (AI) models-integrating diverse data such as imaging and clinical records-are advancing rapidly in healthcare, yet a significant disconnection persists between these complex predictive architectures and the explainable AI (XAI) techniques used to interpret them. We conducted a scoping review over 4 bibliographic databases to investigate the use of explainability methods in cross-modal medical AI studies. From 82 included studies, we found that the landscape remains dominated by independent feature attribution (assigning importance scores to individual modality in isolation), with the majority of studies relying on post-hoc methods (applied after a model decision is reached) that treat the model as a 'black box'. While emerging trends like visual grounding (linking textual justifications directly to specific image regions) and model reasoning show promise, a critical gap remains in explaining the underlying reasoning process. Standardised evaluation is missing in the majority of studies relying solely on qualitative measures. Only a minority of studies achieve good reproducibility with public codebase. We provide suggestions for the field to transition from individual and post-hoc XAIs toward intrinsically explainable designs where the reasoning logic is built directly into the model architecture to ensure that AI outputs align with human-centric clinical workflows and applications.
Kai Hu, Xing-Yue Fu, Yupeng Zhang et al.· npj Digital Medicine· 0 citations
Artificial intelligence has made a great impact on healthcare by providing accurate disease diagnosis, personalised treatment regimens, and efficient clinical decision making. But many of the advanced machine learning and deep learning models are black-box systems, and healthcare professionals find it difficult to understand the logic behind their predictions. This opacity hinders the adoption of intelligent systems in clinical settings where trust and accountability are a must. In this review paper we compare the main interpretability techniques that have been used in clinical decision support systems. These techniques include Local Interpretable Model-Agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), saliency maps, Gradient-weighted Class Activation Mapping (Grad-CAM), attention mechanisms, and decision trees, among others. We performed a systematic literature review to evaluate these techniques based on interpretability, computational complexity, scalability, transparency, and clinical relevance. A systematic literature review was performed to evaluate the techniques in terms of interpretability, computational complexity, scalability, transparency and clinical relevance. The analysis shows that SHAP provides complete local and global explanations, while LIME provides computationally efficient local interpretations. Visualisation based methods such as Grad-CAM and saliency maps are especially useful for medical image analysis, while attention mechanisms are suitable for sequential healthcare data. The study concludes that explainable artificial intelligence improves trust, reliability, and accountability in healthcare systems and is a prerequisite for successful integration of intelligent technologies into clinical practice.
Keywords: machine learning; clinical decision support systems; Explainable Artificial Intelligence; Healthcare Analytics; interpretability
Riya Jacob K· International Journal of Tec...· 0 citations
Although promising, LLM-based systems are not yet reliable enough for autonomous medical diagnosis, and multiple recommendations for future research are contained to ensure a high level of safety, transparency, and clinical applicability for LLMs and other AI/ML-related technologies and devices.
M. U. K. Gunawardhna, Pirunthavi Wijikumar, D. Weerasinghe· Sri Lankan Journal of Applie...· 0 citations
Introduction Artificial intelligence (AI) is reshaping healthcare, enabled by advances in computing, affordable data storage, and the widespread adoption of electronic health records (EHRs). Machine learning (ML), deep learning (DL), and natural language processing (NLP) are increasingly used for disease diagnosis, risk prediction, and treatment planning. Objective This systematic review aimed to examine AI applications across clinical domains from 2020 to 2025, assess their diagnostic accuracy and clinical performance relative to standard practice, identify key implementation barriers including regulatory compliance, algorithmic fairness, and transparency challenges, and compare validation practices and methodological quality with earlier systematic reviews. Methods This systematic review followed PRISMA 2020 guidelines. We searched five databases (PubMed, IEEE Xplore, Web of Science, Springer, and Semantic Scholar) for studies published from January 2020 to September 2025. We included original clinical AI studies that reported prospective validation and/or external validation. Results Twenty studies met the inclusion criteria. Publication volume peaked in 2024 (n = 7, 35.0%). DL approaches were most common (n = 12, 60.0%), with convolutional neural networks (CNNs) frequently applied to medical imaging tasks. By clinical domain, 30.0% of studies focused on radiology (n = 6), 20.0% on oncology (n = 4), and 15.0% on cardiology (n = 3). For imaging-based diagnostic models, the descriptive median performance across individual studies was 0.91 AUC (no formal meta-analysis was conducted due to heterogeneity in study designs, populations, and outcome metrics). The most frequently reported challenges were regulatory compliance (55.0%, n = 11), limited algorithmic transparency (40.0%, n = 8), data quality limitations (35.0%, n = 7), and barriers to clinical integration (30.0%, n = 6). Conclusions AI demonstrates strong potential to improve the effectiveness, safety, and quality of healthcare. However, broader clinical adoption remains constrained by regulatory requirements, interpretability gaps, data quality issues, and workflow integration challenges, underscoring the need for stronger validation practices and more implementation-focused research.
Ghulam Hussain Noori, Shaista Bibi, Seung Won Lee· Inquiry : a journal of medic...· 0 citations
A reproducible experimental approach to developing and evaluating explainable AI systems for healthcare analytics that integrates the steps of data preprocessing, predictive modeling, interpretation generation, and evaluation into one seamless workflow that can be applied to both structured clinical data and medical imaging datasets.
Yashwant Dongre, Deepali A. Godse, Prawit Chumchu et al.· Journal of Visualized Experi...· 0 citations
This study aims to illustrate the significance of Abstract Science Intelligence (XAI) in the medical field, outlining its role in enhancing the interpretability of traditional "black-box" AI models.The goal of this research is to shed light on the significance of Abstract Science Intelligence (XAI) in the medical sector, elucidating its function in bolstering the interpretability of conventional "black-box" Artificial Intelligence models. In medical imaging, disease diagnosis, clinical decision support, and precision medicine, deep learning techniques have proven to be extremely successful; however, the inability to explain and understand how deep learning models work raises a variety of challenges for clinician trust, patient safety, ethical accountability, and regulatory compliance. This review provides a comprehensive overview of recent developments in XAI for healthcare, focusing on key methods for achieving interpretability, such as intrinsically interpretable models, and post-hoc explanation methods such as SHAP, LIME, Grad-CAM, attention mechanisms, surrogate models, and counterfactual explanations. A detailed review of the use of these methods in a variety of clinical areas such as radiology, oncology, cardiology, genomics, electronic health records and drug discovery is also given. Furthermore, the paper examines technical issues concerning explanation fidelity, computational complexity, robustness, scalability, and model validation, as well as ethical issues such as fairness, transparency, privacy, bias, and governance. Other research trends are also discussed, such as causal explainability, humancentered XAI, models that are uncertain, human-in-the-loop systems, and evaluation frameworks. In summary, the review highlights that explainability is not just a technical aspect but a key component in creating AI systems that are both trustworthy and clinically sound and can assist in safe and effective health care decision-making.
Dr. S. Rasheed, Mansoor Ali, Dr .A. Meher Nisha et al.· International Journal of Int...· 0 citations