The Transformer Revolution, Part 1: Dynamic Processing through Output-Weight Interconnections

Abstract

We reinterpret Transformer inference by developing a functionally equivalent mechanical-structural description of its functional architecture. Parameterized transformations of token representations, or transforming concepts, are identified with simple neural networks organized through output-input and output-weight interconnections. This redescription makes explicit an organizational feature that is not equally salient in the standard matrix description: during inference, the outputs of some networks determine the weights, and hence the transformations, of others. These output-weight interconnections generate prompt-dependent dynamic transformations and give rise to Sequence-level Interactive Dynamic Parallel Processing (SIDPP). We show that the number of dynamic parameters grows linearly with prompt length and may become comparable to, or exceed, the number of static parameters fixed through training, a phenomenon we call strong prompt sensitivity. Philosophically, this shifts the conceptual picture of the Transformer from one centered on the static structure acquired through training to one that also treats the prompt-dependent transformations dynamically constructed during inference as constitutive features of its operation. GPT-4.5's recent Turing test results provide a behavioral illustration of this phenomenon. Finally, we identify biological mechanisms morphologically and functionally correspondent to output-weight interconnections, supporting the in-principle neural realizability of SIDPP and motivating Conjecture T: human neural systems may realize a functional architecture relevantly similar to that of the Transformer.