Siddhesh Nadkarni

Query Expansion and Key Specialization in Transformer Attention Geometry

Additional token-level checkpoint analyses show that the monotonic paired-contrast trend is not universal across pretrained families, but survives as an early-training regime that later decays over a full pretraining run, and the link between interaction-rank geometry and attention entropy remains visible in several mo...