TimeHome: Heterogeneous Mixture-of-Experts for Time-Series Foundation Model
TL;DR
TimeHome is a universal sparse transformer basic model for handling heterogeneous time series that incorporates a Heterogeneous Mixture-of-Experts (H-MoE) component, where different expert types are chosen dynamically based on a low-rank temperature-controlled gating mechanism to fit various sequence features.
Abstract
Time-series analysis is important for various scientific and industrial fields, such as remote sensing where observations may be disturbed by clouds, have irregular revisits and different sensors. Current transformer-based approaches have enhanced the modeling of distant time relationships, but they are still constrained by specific task designs and poor adaptability to various time-series patterns. To solve these problems, we present TimeHome, a universal sparse transformer basic model for handling heterogeneous time series. TimeHome incorporates a Heterogeneous Mixture-of-Experts (H-MoE) component, where different expert types are chosen dynamically based on a low-rank temperature-controlled gating mechanism to fit various sequence features. Moreover, a hybrid local–global attention mechanism is designed to consider both short-term variations and long-distance correlations, while specific heads are used for unified prediction, missing value estimation and abnormal event detection. TimeHome is pretrained on TS-200B, a huge database of time series including various temporal patterns from different domains. Comprehensive tests on several benchmark datasets and remote sensing extended evaluations show that TimeHome performs well in long-term prediction, missing value replacement and abnormal event detection. The model also exhibits good zero-shot adaptation ability and fast inference speed by adjusting experts dynamically. The source code and pre-training data will be released publicly.