This work proposes a model-centric Explainable Artificial Intelligence (XAI)-based approach to identify data poisoning in medical imaging classifiers and introduces an explainability-driven paradigm for post-training poisoning detection in medical imaging systems.
Abstract
The integrity of training data is crucial for reliable medical image analysis; however, modern deep learning models remain vulnerable to data poisoning attacks. Such attacks can degrade diagnostic performance while remaining difficult to detect, particularly when a small fraction of the training data is poisoned. Existing defenses require access to the training data or rely on manual data sanitization techniques, which limit their applicability in medical imaging environments constrained by privacy and deployment restrictions. In this work, we propose a model-centric Explainable Artificial Intelligence (XAI)-based approach to identify data poisoning in medical imaging classifiers. The proposed approach examines the consistency between model predictions and associated relevance explanations. Discriminative features extracted from the explainability maps are used to train secondary classifiers that distinguish between models trained on clean and poisoned data. Our approach eliminates the need for data-level inspection and introduces an explainability-driven paradigm for post-training poisoning detection in medical imaging systems. The proposed framework is validated across multiple medical imaging modalities using Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures. Experimental results demonstrate strong data poisoning detection under low poisoning rates, achieving AUC scores of 0.88–0.94 across datasets at a 2% poisoning rate. These results highlight the potential of explainability-driven features as an effective privacy-preserving mechanism for detecting poisoned medical imaging models.
With the rapid development of the cloud computing and Internet of Things (IoT) technologies, the massive deployment of large-scale data processing systems has become possible, especially in the healthcare field where medical image analysis is used. Deep learning models have shown impressive results in diagnostic tasks, but their application in cloud-based systems introduces key privacy and security issues, such as being susceptible to adversarial attacks. Adversarial perturbations can fool classification models, leading to misdiagnosis in medicine, while the sharing and handling of personal patient information can expose the healthcare system to privacy violations. To overcome such challenges, this paper suggests a hybrid secure inference system that combines adversarial example detection with homomorphic encryption-based privacy preservation. The proposed solution is a rather light convolutional neural network (CNN) for the detection of adversarially manipulated inputs and a denoising process to reduce the impact of perturbations prior to the classification stage. The clean or restored images are then secured by means of the CKKS homomorphic encryption scheme, which allows for computing on encrypted data without exposing sensitive information. The images are then encrypted and fed through a deep neural network to classify them in a privacy-preserving manner. Experimental results on a dataset of brain tumor images show the effectiveness of the proposed framework. The model outperforms a baseline CNN model in adversarial and clean conditions with 94.4% classification accuracy, compared with the 71.1% accuracy the baseline CNN model had under adversarial conditions. The results support that the proposed framework has succeeded in providing better adversarial robustness while preserving data privacy, which is acceptable in cloud-IoT environments for secure medical image analysis.
Vijayalakshmi MM, Neelam Malayadri· International journal of com...· 0 citations
Deep Neural Networks (DNNs) deployed in high-risk domains, such as healthcare and autonomous driving, must be not only accurate but also understandable to ensure user trust. In real-world computer vision tasks, these models often operate on complex images containing background noise and are heavily annotated. To make such models explainable, Concept-based Explainable AI (CXAI) methods need to be assessed for their applicability and problem-solving capacity. In this work, we explore CXAI use cases in multi-label classification by training two DNNs, VGG16 and ResNet50, on the 20 most annotated labels in the MS-COCO dataset (Microsoft Common Objects in Context). We apply two CXAI methods, CRP (Concept Relevance Propagation) and CRAFT (Concept Recursive Activation FacTorization), to generate concept-level explanations and investigate the overall evaluations. Our analysis reveals three key findings: (1) CXAI highlights learning weaknesses in DNNs, (2) higher concept distinctiveness reduces label and concept confusion, and (3) environmental concepts expose dataset-induced biases. Our results demonstrate the potential of CXAI to enhance the understanding of model generalizability and to diagnose bias instigated by the dataset.
Pneumonia remains a major cause of lung disease globally, and its timely and reliable diagnosis is crucial. Radiology is often used to detect the infection in a chest X-ray, but this process can be laborious and depend on the radiologist’s interpretation. However, recent advances in deep learning techniques have demonstrated high accuracy in automated pneumonia detection, but their "black-box" nature hampers their practical use. Health-care professionals often need to understand the reasons behind predictions to trust the automated system. This study introduces a machine learning-based approach with additional explainable techniques to enhance model performance and explainability in predicting pneumonia. The proposed method uses a convolutional neural network to predict chest X-ray images, and explainability techniques like Grad-CAM and SHAP are used to explain which parts of the image contribute most to the prediction. We test the system on the RSNA Pneumonia Detection Challenge Dataset that includes expert-provided infection labels. The proposed explanation method is evaluated by comparing the model’s predictions with expert annotations. This study demonstrates that our system not only provides accurate classification results but also provides plausible visual explanations that correspond to the desired locations in the chest. This approach can help boost trust in AI-driven medical diagnostic systems and enable their potential deployment in clinical practice.
Corrupted, inconsistent, or anomalous data silently threatens the safety and reliability of medical AI. Despite growing regulatory recognition of dataset quality assurance (QA) for high-risk medical AI, scalable automated detection remains underdeveloped. We employ unsupervised anomaly detection (AD) and out-of-distribution (OOD) detection as an automated dataset QA mechanism for multi-center dynamic contrast-enhanced breast MRI. We build a controlled AD benchmark of 17 realistic QA-relevant anomaly types from six public datasets (protocol violations, processing errors, incorrect anatomical regions) and propose a taxonomy of radiological image anomalies based on human visual perception, enabling fine-grained analysis of AD failure modes. The benchmark includes near-, medium-far-, far-OOD samples, as well as in-distribution and external normal data. Four methods are evaluated: a projection-based method extended with a domain-specific feature extractor and a novel positional encoding, a reconstruction-based approach extended to full 3D volumes with an augmented training objective, and two unmodified hybrid OOD detection methods. Medium-far- and far-OOD samples are detected reliably, whereas near-OOD samples and external normal data from unseen institutions expose method-specific differences. The 3D reconstruction-based approach best balances detection performance (AUROC: 0.936) and generalization to unseen institutions. The projection-based method with positional encoding achieves the highest overall detection performance (AUROC: 0.954). Both hybrid methods exhibit critical failure modes, confirming that methods validated for one modality or anatomy may not generalize without domain-specific adaptation. Implants and mastectomies remain an open challenge for all methods. Our results establish a foundation and practical guidance on scalable unsupervised QA in medical AI pipelines.
C. Tappermann, Steffen Renisch, L. O. Schwen et al.· 0 citations
This work proposes FedMHDet: Model Hint Federated Learning Detection Model, a novel federated learning detection framework that leverages multi-scale feature consistency as a global model hint to guide client models, thus mitigating the feature drift problem.
Zhenghua Xu, Gaoxi Zhou, Hexiang Zhang et al.· IEEE journal of biomedical a...· 0 citations
Background Chest X-ray is the standard screening tool for pulmonary diseases, enabling early detection and timely intervention of lung lesions. Due to scarce and costly abnormal labeling of chest X-rays, unsupervised anomaly detection trained only with normal images has become a major research focus, especially, reconstruction-based methods are favored for learning normal data patterns and making it easier to visualize abnormal regions. Although the current reconstruction-based methods of chest X-rays have achieved good result, they have three problems. Firstly, over-reliance on pixel-level alignment in the spatial domain hinders the capture of global information. Secondly, single-layer feature comparison between original and reconstructed images lacks multi-layer and multi-image analysis, resulting in poor feature consistency. Thirdly, using reconstruction error as anomaly score struggles to balance sensitivity to anomalies and fidelity to normal. Using a reconstruction-based approach trained only on normal images, this study improves rapid screening of thoracic abnormalities and shifts screening criteria toward earlier detection of chest lesions. Methods We propose an architecture DualA-AD including Dual-domain Adversarial Learning (DAL), Feature Constraint Module (FCM) and Distributed Anomaly Score (DAS). In the training stage, DAL captures image details and global information by integrating spatial and frequency domains for dual-domain adversarial reconstruction. While FCM improves feature consistency between reconstructed and original image by analyzing multi-layer feature difference within individual images and similarity differences across multiple images. In the testing stage, DAS enhances image discriminability by fitting a discriminator’s output on the training set as the real distribution and quantifying the deviation of images from this distribution. On three mainstream public chest X-ray datasets, the model was trained using only normal chest X-rays and evaluated on their respective test sets. Result In the experiments on three datasets, DualA-AD achieved the best AUC, ACC, and F1 scores against classical, general, and specialized baselines, with AUC improvements over the SOTA of 0.93, 0.77, and 0.48%, respectively. Conclusion DualA-AD improves the accuracy for distinguishing normal and abnormal chest X-rays. The method provides an efficient scheme for fast, low-cost intelligent detection of lung abnormalities, with great potential for computer-aided chest radiographic diagnosis and early prevention and management of pulmonary disorders.
Na Liang, Yehong Tong, Xingyu Zhang et al.· Frontiers in Medicine· 0 citations
