Skip to content

Loss-Guided Pretraining Data Selection for Time-Series Foundation Models

Sep 2026 · arXiv.org · Vol abs/2609.37255 · 0 citations · 27 references
Computer Science

TL;DR

This work connects forecasting loss to optimization difficulty by showing that normalized squared loss controls the per-sample gradient norm under a local Jacobian condition and defines a reference loss score and applies dataset-stratified selection to preserve the diversity of samples.

Abstract

Time series foundation models (TSFMs) are pretrained on heterogeneous collections containing billions of observations, yet their training windows are typically sampled without estimating whether they provide useful learning signal. We introduce a static data-selection framework that scores each window with a reference forecaster and retains an intermediate interval within every source dataset. Specifically, we connect forecasting loss to optimization difficulty by showing that normalized squared loss controls the per-sample gradient norm under a local Jacobian condition. We then define a reference loss score and apply dataset-stratified selection to preserve the diversity of samples. Across various TSFM architectures, our method outperforms random selection by an absolute margin and even improves both relative MASE and CRPS over full-data pretraining by retaining fewer candidate pretraining windows. Further analyses show strong cross-scale and cross-architecture score correlations, indicating that a small reference model can often select data for larger targets, provided that the reference and target share compatible difficulty orderings.

View source

Similar papers

Aurora-X: Built for Extreme Time Series Forecasting

A novel pattern-guided mixture-of-experts that expands model capacity through sparse activation and uses shallow patch similarities to constrain deep-layer routing, guiding expert specialization across heterogeneous time series and an implicit quantile network head that predicts arbitrary quantiles to characterize pred...

Xingjian Wu, Chen-Juan Guo, Xiangfei Qiu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Does Joint-Embedding Predictive Architecture Pretraining Help Time Series Forecasting?

A large scale evaluation of one JEPA instantiation across nine backbones and eleven benchmarks spanning temporal and spatio-temporal forecasting finds that the benefit of this instantiation varies sharply across backbones, indicating the variability is a general property of this instantiation rather than a dataset spec...

Yu-Tong Feng, Bo-Wen Liao, See-Kiong Ng et al. · 0 citations

When, Not How Much: Evaluating Time-Series Foundation Models on Sparse Events

Pretrained time-series foundation models (TSFMs) are evaluated as forecasters of future values, yet for sparse series many decisions depend only on which future periods contain activity. Standard benchmarks do not assess this. On five sparse datasets, we rank positions within forecast windows that contain both events a...

Daniel Schoess, F. von Wangenheim · 0 citations
#artificial intelligence Preprint Oct 2026

AdaCast: Conditional Parameter Generation for Adaptive Time Series Forecasting

Time-series foundation models (TSFMs) have achieved strong forecasting performance across domains. However, most adaptation methods remain static. Existing all-in-one methods learn a single set of dataset-level parameter updates and apply the same adapted model to every input. As a result, they cannot adapt the model p...

Darahaas Nallagatla, Darryl C. Jacob, Pan He · 0 citations
#machine learning Preprint Sep 2026

Tabby: An Open Pretraining Recipe for Time Series Foundation Models

Tabby, a long context probabilistic time series foundation model, is released together with a complete and open recipe of how it was built, which achieves competitive zero-shot forecasting performance on GIFT-Eval and the out-of-distribution TIME benchmark.

Shi-Feng Xie, Bahaeddine Abdessalem, Ze-Hao Xiao et al. · 1 citation
#machine learning Preprint Oct 2026

GRAM: Correcting Frozen Time-Series Foundation Models via Graph-Retrieved Amplitude Memory

Time-series foundation models (TSFMs) enable zero-shot forecasting through large-scale cross-domain pretraining, while retrieval augmentation further improves their performance by leveraging historical information. However, existing methods typically correct TSFM forecasts using the ground-truth futures of similar hist...

Xiao-Yun Yu, Xiangfei Qiu, Yong-Gui Huang et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.