Measuring the Predictive Horizon: How Far Can Local Explanations Forecast Autoregressive Language Model Behavior?

Abstract

Mechanistic interpretability aims to reverse-engineer Large Language Models (LLMs) into human-understandable components, assuming that these components can predict and control model behavior. We challenge this assumption by demonstrating that autoregressive generation exhibits chaotic dynamics and Rashomon multiplicity. Drawing on dynamical systems theory, we provide a theoretical framework suggesting that mutual information between an explanation at token t and model behavior at token t + k decays exponentially. We term this limit the Explainability Horizon (HE). We offer an empirical characterization of this phenomenon, measuring a strict threshold-based horizon of 8–12 tokens and a visual noise floor around 15–20 tokens across models ranging from 117M to 8B parameters. We validate this horizon against human judgments of safety, finding strong alignment (ρ = 0.74) with proxy metrics. We further analyze how explanation fidelity degrades with token distance, how attribution stability behaves under prompt perturbation and long-context conditions, and how hallucination-related uncertainty propagates across longer forecast horizons. Our findings suggest that while interpretability is valuable for local debugging, it cannot, at the scales studied here, reliably forecast long-horizon safety without complementary trajectory-level monitoring.