Do Sophisticated Forecasting Models Beat Naive Baselines? A Reproducible Benchmark with Statistical Significance Testing

Abstract

Time-series forecasting is a foundational task in data science, yet reported improvements over naive baselines are frequently small, dataset-dependent, and rarely subjected to formal significance testing. We present a reproducible benchmark comparing three families of forecasting models — naive baselines, classical statistical models (ARIMA, Holt-Winters, SARIMA), and machine learning models (Random Forest, Gradient Boosting) — on three standard datasets: AirPassengers, Sunspots, and Mauna Loa CO2. Each model is evaluated under an identical 80/20 chronological split with fixed random seed 42, and paired Wilcoxon signed-rank tests are used to assess statistical significance of the differences in absolute errors. Our findings reveal a nuanced picture: SARIMA significantly outperforms the LastValue baseline on AirPassengers (p<0.001, ~40% RMSE improvement), and tree-based ML models significantly outperform the baseline on Sunspots (p<0.05). However, on CO2, no sophisticated model beats the LastValue baseline, and all differences are significant in the wrong direction (p<10^-68). An ablation study shows that the optimal number of lag features varies by dataset from 1 (CO2) to 24 (AirPassengers), suggesting that feature engineering requirements depend strongly on the seasonal structure of the data. We argue that (a) naive baselines must be reported in any forecasting benchmark, (b) statistical significance testing should be standard, and (c) model selection should be guided by data characteristics rather than complexity.