How Language Models Differ in Redistributing Attention-Head Activity Under Serial Demand
The way a model distributes activity over each layer's attention heads offers a coarse view of how it routes information through depth; how this changes with the task is part of what a mechanistic account must explain. Holding prompt length fixed, we vary how many serial steps a task demands and measure, in every layer...
Johnny Jingze Li, Abdulla Kuleib, Kalyan S. Basu et al.
· 0 citations