Evaluating Neural Probabilistic Forecasts of Sparse Earthquake Counts: What Survives a Careful Evaluation

Abstract

We revisit a neural negative binomial model for weekly earthquake counts on a 3 x 3 degree grid over the Tian Shan and Pamir, and ask which of its conclusions survive a careful evaluation. At the completeness magnitude of 4.5, the 144-week test period holds 2448 cell-weeks; six cell-weeks, in five distinct weeks, have three or more events. Two conclusions survive. First, Poisson forecasts, temporal ETAS included, understate the probability of such a busy week: ETAS expects 0.72 where six occurred, a shortfall by a factor of 8.3 (p = 0.00009, exact test). One dispersion parameter removes it (ETAS with dispersion expects 4.45, p = 0.58); simulating aftershock chains within the week also reproduces the tail but overshoots the total count by about a quarter. Second, the network's advantage over a linear model disappears once that model has an intercept per cell: the remaining gain is 3.4% of deviance, not significant (p = 0.57 over five test years), and it comes from one cell-week out of the 4420 scored; without it the network is 0.7% worse. A blind test on 2024 to September 2026, with every model fitted on data through 2023, confirms the tail result: Poisson forecasts are rejected (p from 3 x 10^-8 to 0.001), dispersion models are not (p between 0.25 and 0.84), except the NB GLM, whose level blew up out of sample and which is rejected at p = 0.003. Out of sample, model families differ in how they extrapolate beyond the training range, not in the per-cell term. We also measure the evaluation traps met on the way: a biased reference level, a single seed (five of six effects in v1's results table change sign), too few bootstrap resamples, and a threshold below completeness, which shrinks the ETAS shortfall from 8.3 to 2.8 times. This version replaces preprint v1; Section 7 lists the corrections.