From Black Box to Clarity: A Systematic Review of Explainability Methods in Deep Convolutional Neural Networks
TL;DR
A PRISMA-compliant systematic review of 154 studies published between 2017 and 2025 found that out of the 154 studies in the review, 98 used objective methods to evaluate fidelity, stability, or sensitivity, and it was found that there is a huge bias in the field regarding post hoc visual methods.
Abstract
Deep neural networks (DNNs) have significantly advanced machine perception and reasoning; however, their lack of transparency in decision-making continues to pose a major challenge, particularly in high-stakes domains such as healthcare, finance, and law. This is especially concerning with the black-box nature of convolutional neural networks (CNNs), where the rationale for making a decision can be as important as the decision itself. This paper is driven by a question that is easier to ask than to answer: how can CNNs be made to explain themselves? To answer the question, we wrote a PRISMA-compliant systematic review of 154 studies published between 2017 and 2025. These studies were selected from 4421 studies retrieved through Web of Science, Scopus, IEEE Xplore, and ACM Digital Library. CNN-specific taxonomy was developed. This taxonomy organizes explainable artificial intelligence (XAI) methods on four axes: explanation timing, model dependency, output type, and target component. We found that there is a huge bias in the field regarding post hoc visual methods. Grad-CAM is the most widely cited visual explanation methodology, and within the model-agnostic framework, LIME and SHAP prevail. This research was also the first to analyze standard assessment methods. It was found that out of the 154 studies in the review, 98 used objective methods to evaluate fidelity, stability, or sensitivity. Conversely, fewer than ten of them used human-centered methods to evaluate how tasks were performed, how the users trusted the method, or how the users were prepared to interact with the system. We argue for a dual-reporting convention under which metrics should be reported together at least once, as per the family of metrics. The third contribution is an evidence-based challenge map, where we outline four issues: absence of standardized benchmarks, post hoc mechanism scalability limitations, vulnerability to adversarial perturbations, and the persistent gap between the technical descriptions and human understanding. For each challenge, we propose concrete directions: integrating causal reasoning, adopting participatory evaluation design, and building hybrid transparent architectures. We offer this review as a practical roadmap for researchers and practitioners working toward more explainable deep neural networks.