Textual World Models for Agent Security: Modelling the Internal State of an LLM Agent from Text Alone

Abstract

An AI agent chooses its own path to reach an objective, so its security and compliance depend on its internal state, not only on its inputs and outputs. We ask whether the activations of the agent's large language model (LLM), produced when it reads the context, are a sufficient observation of this state to build a model of it, without any access to the agent's tools or design. We test sufficiency by prediction: a world model built on these activations should predict the agent's next step. Because the agent's LLM is usually not accessible, a small witness LLM reads the same context. On 22 agents, the resulting model predicts that the next answer will carry out a prompt-injection attack with an AUROC of 0.93, and identifies the conversation in which the agent will give in for 63 % of pairs that a lookup table cannot separate. The observation is partial, as expected: nothing predicts an attack before it enters the context, because that part of the state is held by the tools.