When Should Forecasting Models Listen to Text? Counterfactual Utility Routing for Multimodal Time Series Forecasting
Abstract
Time-series forecasting supports decisions in domains such as finance and healthcare, where forecasts may depend on events not fully represented in numerical histories. Multimodal forecasting can supply this context through text, but the additional information is useful only when it remains relevant to the target and forecast horizon. Existing methods commonly fuse text at every prediction step, even when it is stale, weakly related to the target, or informative only for certain horizons. We therefore frame text use as a horizon-wise decision and introduce CUTS, a Counterfactual Utility Routing framework for selective fusion. For each forecast step, CUTS retrieves horizon-specific evidence and produces matched text-conditioned and no-text forecasts. A utility estimator predicts the benefit of the retrieved evidence, while its uncertainty moderates the final interpolation between the two forecasts. On Time-MMD, CUTS achieves the best nine-domain normalized MSE and MAE among the evaluated methods. It records the lowest MSE in five of nine domains. On FNSPID, it achieves the lowest average MSE and MAE and obtains the lowest MSE at the 3-day and 5-day horizons. Ablations identify horizon-specific evidence selection and staged training as key contributors. Corruption and routing analyses further show that uncertainty-aware routing helps CUTS limit its reliance on unreliable text.