A Novel Tokenization-Free GPT-2 Mechanism for Intermittent Demand Forecasting

Abstract

Intermittent demand — spare parts, slow-moving SKUs, long-tail retail — has been one of forecasting’s oldest open problems, and the recent wave of LLM-based forecasters has not yet settled the picture. Across the literature, the design choices that shape an LLM forecaster — backbone, output head, loss function, input representation, training scope — co-vary across studies, and the effect of any single choice on accuracy is rarely isolated. We run a controlled comparative assessment within the GPT-2 family on 384 stratified M5 intermittent series, crossing two output-head architectures (linear projection vs. two-layer MLP) with two loss functions (compound BCE+MSE vs. pure count NLL), holding the backbone, input representation, tuning protocol, and evaluation metric fixed. The head-architecture contrast is statistically indistinguishable from null under both losses; switching from compound BCE+MSE to pure count NLL increases per-series mean RMSSE by roughly 18% under both heads, with the penalty concentrated in the highest-intermittency strata where non-zero observations are rarest. Benchmarked against LSTM, LightGBM, and six classical Croston-style methods, the compound-loss GPT-2 configurations sit within the Nemenyi critical difference of LSTM at the headline 28-day horizon and track it closely across the reported horizons. For per-series intermittent demand forecasting with pretrained transformers, the supervision signal matters far more than the architecture of the head sitting on top of it; the design budget should reflect that. An external validation on 100 synthetic series with controlled intermittency, under replication criteria fixed in advance, reproduces the loss-function effect at matching magnitude (+17%) and shows the penalty rising monotonically with the average inter-demand interval.