Christos Thrampoulidis

Structure Before Collapse: Transient semantic geometry in next-token prediction

This work finds that semantic geometry emerges early in training, and that representations cluster by shared attributes despite receiving no explicit supervision to do so, and proposes a preliminary modification to the commonly used unconstrained features model to capture the emergent semantic geometry.