Skip to content

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Sep 2026 · arXiv.org · Vol abs/2609.28876 · 2 citations · 39 references
Computer Science

Abstract

Forecasting requires large language model (LLM) agents to gather evolving evidence, revise their beliefs, and make calibrated predictions over time. Yet evaluating and improving these capabilities is challenging because real-world events unfold continuously and outcomes may take months to resolve. We introduce Forecast-Dojo, a replayable environment that reconstructs resolved prediction-market events as sequences of dated forecasting steps, enabling reproducible evaluation, longitudinal analysis, and training from agent interactions. It comprises 1,568 Polymarket events split temporally into training and evaluation sets, alongside 18.8 million dated news articles. Across 12 models, research tools consistently improve forecasting performance, and greater capture of new evidence is associated with larger improvements over time. Nevertheless, all models trail historical market forecasts, while carrying a belief notebook across dates reduces research costs without consistently improving forecast quality. We supervise fine-tuned Qwen3-30B-A3B-Thinking-2507 on collected trajectories from Qwen3-235B-A22B-Thinking-2507 to demonstrate the environment's training utility. The student's accuracy is improved from 34.8% to 42.8% on the evaluation set.

View source

Similar papers

#machine learning Preprint Sep 2026

From Narrative to Auditable Forecasts: A Structured Scaffold for Agentic Forecasting

Auditable forecasting report that makes forecast construction explicit and supports systematic post hoc analysis, and improves forecasting accuracy and calibration relative to strong agentic baselines, surpasses market-implied references in several settings, and outperforms substantially more expensive deep-research ag...

Yuan-Pu Cao, Yongkang Du, Yu-Rui Chang et al. · 0 citations
#machine learning Preprint Oct 2026

Do Your Own Research: Learning to Forecast by Learning to Search

An agentic forecasting environment, dataset, and harness built from 2,100+ resolved Polymarket questions is introduced; the agent acquires its own context at rollout time (web search, page reading, and financial time series, all restricted by layered leak filtering to information published before each question's cutoff...

Yusuf Afifi, Artur Kiulian, Anton Polishko et al. · 0 citations
#machine learning Preprint Oct 2026

Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target pol...

Angel Wang, Dominique C. Perrault-Joncas, Alvaro Maggiar et al. · 0 citations
#artificial intelligence Preprint Sep 2026

LEAP: Likelihood Elicitation and Aggregation for LLM-based Probabilistic Forecasting

This work proposes LEAP (Likelihood Elicitation and Aggregation for Probabilistic forecasting), which reorganizes how collected evidence is used in the prediction stage and improves most prediction and calibration metrics across models and remains stronger under controlled comparisons of prior access, inference budget,...

Yu-Fei Chen, Yi-Ran Zhao, Xiao-Gang Xu et al. · 2 citations
#machine learning Preprint Sep 2026

Downside-Controlled Online Forecast Combination under Delayed and Revised Outcomes

Post-hoc correction adjusts a forecaster that cannot be retrained, such as a foundation model, but a correction fitted where errors are stable can hurt where they shift. We aim for downside control: not much worse than the starting forecast. We combine the frozen forecaster, a static corrector and an online corrector o...

Minkyoung Kim, Hyunjung Byun, Yo-Han Lee et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.