A quantitative benchmark for explainable artificial intelligence in astronomical transient classification

Abstract

Explainable artificial intelligence ( , an open benchmark that evaluates explanation quality along five complementary axes -- perturbation-based faithfulness, deletion and insertion AUC, complexity, cross-representation plausibility, and the standard model- and data-randomisation sanity checks -- applied to nine classifiers spanning tree-based, deep-learning, and physics-informed architectures on the methods are increasingly used to interpret machine learning classifiers of astronomical transients, yet their reliability is rarely assessed quantitatively. We present astro-xai-bench photometric transient dataset (14 classes, six passbands), with all metrics averaged over five random seeds. A detailed case study of XGBoost (tree-based) and CNN2D (convolutional on Gramian angular field images) illustrates how each axis interacts with model architecture: faithfulness is representation-dependent, complexity rankings invert under basis change, and projected plausibility is sensitive to the projection rather than the model. Three general findings emerge: (i) deep-learning models on raw light curves align better with an astrophysically informed reference mask than tree-based models trained directly on the hand-crafted features that define the mask; (ii) the sanity checks reveal that LSTM, CNN1D, and Random Forest attributions are dominated by the input distribution, while XGBoost, CNN2D, ViT, and both physics-informed baselines (Avocado, ParSNIP) pass all applicable checks; (iii) no single model dominates all five axes, making multi-axis evaluation essential. All code and evaluation pipelines are released as a public benchmark.