Climate Knowledge in Large Language Models

Abstract

Large language models (LLMs) are increasingly deployed for climate-related applications, where understanding internal climatological knowledge is important for assessing their reliability. Despite growing adoption, the capacity of LLMs to recall climate normals from parametric knowledge (information stored in the model's weights, without external retrieval) remains largely uncharacterized. We investigate this capacity focusing on a prototypical query: mean July 2-m air temperature 1991-2020 at specified locations. We construct a global grid of queries at 1° resolution land points, providing coordinates and location descriptors, and validate responses against ERA5 reanalysis. We further extend the evaluation to the twelve monthly normals for four of our best-performing LLMs on a subset of the grid. Results show that LLMs reproduce broad geographic regularities in climatological temperature fields, capturing latitudinal and topographic patterns, with July root-mean-square errors of 3-6 °C and biases of ±1 °C; errors are comparable across the annual cycle. However, spatially coherent errors remain: performance degrades sharply above 1500 m, where RMSE reaches 5-13 °C compared to 2-4 °C at lower elevations. Including geographic context (country, city, region) reduces errors by 27% on average. While models capture the global mean magnitude of observed warming between 1950-1974 and 2000-2024, they fail to reproduce spatial patterns of temperature change. This limitation highlights that while LLMs may capture present-day climate distributions, they struggle to represent the regional and local expression of long-term shifts in temperature essential for understanding climate dynamics. Our evaluation framework provides a reproducible benchmark for quantifying parametric climate knowledge in LLMs.