Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting
TL;DR
A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters are selected based on line-level forecast accuracy, a gap that can be reduced without retraining.
Abstract
A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters are selected based on line-level forecast accuracy. This disconnect matters when demand is intermittent and lumpy, histories are short, and lead times span months. We benchmarked 38 forecasting methods spanning classical, intermittent-demand, machine-learning, deep-learning, and pretrained foundation models. A common decision-aware protocol evaluates them on an industrial panel drawn from a live contract and two public datasets. Forecast-accuracy rank and order-service rank are negatively correlated on the industrial panel, at -0.555, across methods evaluated on 20,330 real multi-item orders. Service is more closely associated with the direction of cumulative forecast bias, including over-prediction during zero-demand periods, than with point accuracy. Examining bias in Chronos-2's instance normalization yields a training-free correction that lifts the per-material fill proxy from 77.5% to 92.0% (14.5 percentage points) at the 90% policy target and raises the complete-order fill rate from 54% to 63%. For reproducibility, we release RUF (Regenerate-Until-Fidelity), a method for generating fidelity-certified synthetic panels on which the findings reproduce. For intermittent demand, the lowest-error forecast need not deliver the highest service. Bias direction helps explain this gap, which can be reduced without retraining.
