The Anatomy of a Truth Direction: Knowledge-Dependent Dimensionality, a Relational Law, and a Shared Category Geometry in Small Language Models
TL;DR
This work conducts extensive evaluation across 14 models from 6 diverse architectural families, read and extract at cost $O(d)$ per token, and makes a pre-registered prediction on whether the arrangement extends to categories whose truth is computed rather than retrieved.
Abstract
B\"urger et al.\ (2024) demonstrated that truth representations in large language models are universal across statement polarity but reside within a multidimensional subspace. The truth value of a statement is linearly readable from a residual stream of language model, but it is not clear how much of that representation fits on a single direction, which component builds it, or what it is made of. We conducted a study based on these questions, with one instrument: a training-free axis, the dominant direction of the singular value decomposition (SVD) of hidden-state differences over true/false minimal pairs, identified without labels up to one global sign. Extensive evaluation across 14 models from 6 diverse architectural families (including MoE), read and extract at cost $O(d)$ per token. We close with a pre-registered prediction on whether the arrangement extends to categories whose truth is computed rather than retrieved.