Learning the world: A Critical Survey of World Models for Physical AI

Abstract

A world model is a learned system that predicts what will happen next under a candidate action, given what an agent has observed and done. With one, an agent can compare actions before it commits, practice skills in imagination and notice when the world surprises it. The idea dates to 1990 and now sits at the center of physical AI. Learned world models render interactive worlds in real time, run in operational weather forecasting and generate sensor data for autonomous driving. The same growth has fragmented the field. Recurrent, transformer, diffusion, state space, joint-embedding and object-centric models are all called world models, each family has its own benchmarks and vocabulary, and the best-looking models are often the worst at following actions. This survey argues that one design choice explains much of this disorder: what a world model’s loss is computed against decides what the model can be used for, how it fails and how it can be measured. A model is generative when that loss lives in observation space, predictive when it lives in a learned embedding, a given physical state or task quantities, and a bridge case when it trains a decoder but acts only in latent space. We begin with what world models are for. Reading driving, robotics, science, healthcare and games through the target each domain has settled on, we find that learned models are in routine use mainly where they are trained against a measured state, as in weather and protein structure. We then map the design space through eight lenses, with the prediction target as the primary one, and examine generative and predictive models in depth. The target also decides which planners a model supports, from learning in imagination to search and model predictive control, and which evaluation questions can be asked of it. Visual realism is close to saturation at short horizons, but physics is not. On tests where appearance cannot help, the best models of both kinds trail human or real-video references by 39 to 71 points. We close by recasting open problems, including error accumulation, counterfactual reasoning and weak action conditioning, as questions about representation, and weigh the evidence for and against the thesis. We also propose a reporting standard for evaluation and a staged research roadmap whose items each name the evidence that would show progress.