The Smallest Singular Directions of the Unembedding Matter Disproportionately for Next-Token Prediction

Abstract

Interpreting, editing and compressing language models often involve assessing which parts of their internal representations matter for prediction. Many of these assessments rely on distance, such as reconstruction error or the length of an intervention. For next-token prediction, however, the logit differences that matter most are those among the few most plausible tokens. These differences can be small. We therefore test whether changes to the final hidden state that move the logits less also change the prediction less. Specifically, changes along the smallest singular directions of the unembedding, which maps this state to the logits, move the logits less than random changes of the same length. In open-weight language models across several families, their prediction change and their change in the next-token loss are nevertheless on average several times those of random changes. Across all singular directions, the prediction change is out of proportion to the logit change only at the smallest. These directions carry a large share of the difference between the model's leading candidate tokens. Consequently, distance-based methods undervalue these directions, and rankings by singular value place them last, although they matter disproportionately for prediction. More broadly, our study highlights that comparing internal representations by distance can miss differences that change what the model predicts.