Graph-Based Features for Late-Delivery Risk Prediction in Supply Chains: A Leakage-Controlled Benchmark
TL;DR
This study asks whether hand-crafted graph features, appended to a tabular XGBoost model, improve late-delivery risk prediction once leakage is controlled, and recommends order-grouped, temporal evaluation with a rule baseline.
Abstract
Predicting whether an order will be delivered late is a core task in supply chain analytics, and the public DataCo Smart Supply Chain dataset is widely used as a benchmark for it. Graph-based features and models are increasingly proposed to improve such predictions, but the dataset contains columns that leak the label and rows that share an order, which can inflate reported performance. This study asks whether hand-crafted graph features, appended to a tabular XGBoost model, improve late-delivery risk prediction once leakage is controlled. We audit the dataset, remove cancelled and fraud orders, and collapse 180,519 item rows into 62,897 orders. We build a heterogeneous graph of customers, products, categories, cities, countries and regions (24,268 nodes, 165,059 edges) with NetworkX, derive structural features, spectral embeddings and fold-safe label-aware encodings (a form of target encoding over graph neighbours), and evaluate them with XGBoost under order-level repeated cross-validation, a temporal split, Holm-corrected paired tests and SHAP. Leakage matters: a leaky column gives an AUC of 1.00, and a random row split overstates AUC by 0.024 (0.7985 versus 0.7749). Without leakage, the label is governed almost entirely by shipping mode and, for Same Day orders, by whether the order is placed after noon. A four-line lookup rule reaches an AUC of 0.774, and Standard and Second Class outcomes are indistinguishable from chance for every model (within-mode AUC about 0.50). Graph features add a nominal 0.003 AUC over XGBoost (uncorrected p = 0.049) that does not survive Holm correction (p = 0.146), cannot be distinguished from the rule (0.001, p = 0.445), and reverses on a temporal split (-0.005 against the rule). Entity-level late rates do not differ from shuffled labels. On DataCo, hand-crafted graph features therefore add no reliable signal; end-to-end graph neural networks were not tested. The whole graph pipeline costs about 5 s and under 1 GB of memory, so the problem is absence of signal, not cost. We recommend order-grouped, temporal evaluation with a rule baseline.