Demystifying black-box machine learning in supply chain risk management: from prediction to intervention via explainable AI

Abstract

This study aims to develop a temporally validated and explainable machine-learning framework for predicting late-delivery risk in supply chain operations. It addresses the tension between predictive accuracy and managerial interpretability by using only pre-shipment operational information and by providing feature-level explanations for risk alerts. The study supports proactive supply chain risk management by showing not only whether an order is likely to be delayed, but also which operational factors contribute to that prediction. The study uses the DataCo Smart Supply Chain data set, comprising 180,519 transaction-level observations from 2015 to 2018. Logistic regression, random forest and XGBoost are compared after removing outcome-based leakage variables. Orders are sorted chronologically so that models are trained on earlier observations and tested on a later temporal holdout period. XGBoost is tuned using TimeSeriesSplit. Class imbalance is addressed through class weighting and scale-position weighting. Robustness is assessed through expanding time windows and business subgroup analysis. Shapley additive explanations (SHAP) and permutation importance are used for interpretability. The temporal holdout results show that XGBoost achieves receiver operating characteristic area under the curve (ROC-AUC) of 0.7414 and precision-recall area under the curve (PR-AUC) of 0.8097, with the strongest default-threshold precision of 0.8407 and specificity of 0.8772. Threshold tuning increases XGBoost recall to 0.8972, making it useful for early-warning scenarios where missed delays are costly. Robustness tests produce stable ROC-AUC values between 0.7301 and 0.7464. SHAP and permutation importance show that shipment timing, customer/order geography, shipping mode and transaction type are stronger delay-risk drivers than financial variables. The study is limited to one public transaction-level data set and does not include real-time external disruption factors such as weather, infrastructure failure, port congestion or geopolitical events. SHAP explains model behaviour rather than causal effects. Future research should validate the framework on additional supply chain data sets, integrate live operational and environmental data and compare SHAP with counterfactual or actionable-recourse methods to support intervention design. The framework enables logistics managers to identify high-risk orders before delivery completion and to understand the operational reasons behind each alert. It can support decisions such as adjusting shipment modes, extending delivery buffers, prioritizing route monitoring and allocating escalation resources. Threshold tuning allows firms to adapt the model to different intervention costs and service-risk priorities. More reliable late-delivery prediction can improve customer service, reduce avoidable disruption and support more efficient logistics planning. By helping firms intervene earlier, the framework may reduce unnecessary emergency shipments, wasted resources and avoidable operational inefficiencies. The study contributes a reproducible decision-support framework that combines leakage-safe prediction, chronological validation, robustness testing and explainable AI. Rather than relying on random splitting or opaque accuracy claims, it evaluates late-delivery prediction under temporally realistic conditions and links SHAP-based explanations to operational intervention.