Do Language Models Update their Forecasts with New Information?
TL;DR
It is found that while models often track the correct update direction, their quantitative shifts remain heavily under-responsive and both verbalized and logits-based confidence estimates are poorly calibrated compared to the human reference standard.
Abstract
Prior work has largely treated future event prediction as a static task, failing to consider how forecasts should evolve as new evidence emerges. To address this gap, we introduce EvolveCast, a framework for evaluating whether large language models (LLMs) appropriately revise their predictions in response to new information. In particular, EvolveCast assesses whether models adjust their forecasts when presented with evidence released after their training cutoff, using human forecasters as a comparative reference for prediction shifts and confidence calibration. We find that while models often track the correct update direction, their quantitative shifts remain heavily under-responsive. Furthermore, both verbalized and logits-based confidence estimates are poorly calibrated compared to the human reference standard. These findings suggest current LLMs are fundamentally limited in their response to new evidence; models treat new information as mere retrieval context rather than as variables that dynamically shift posterior probabilities. EvolveCast highlights the critical need for robust mechanisms to incorporate external knowledge into dynamic belief updating.
