Morphological Hijacking in Frozen Language Models: A Contrastive, Symmetry-Regularized Projection Head for Algebraic Structure Recovery

Abstract

Preprint. Not yet peer-reviewed. Abstract: Frozen autoregressive language models cluster surface-similar tokens together even when a stronger, task-relevant structure is available in the input. We construct an adversarial lexicon (LUXVAR Core-30) in which word meaning is defined by orbit membership under the finite algebraic group Aff(Z12), while a long, orthographically salient prefix is deliberately uncorrelated with that meaning. Across four frozen backbones (GPT2-small, Pythia-410m, Mistral-7B-v0.3, Qwen2.5-7B), unsupervised clustering of final-layer hidden states recovers the surface prefix almost perfectly (Prefix Hijacking Ratio = 1.000) while failing to recover the underlying orbit structure (Adjusted Rand Index ≈ −0.148 on all four models) — a phenomenon we term morphological hijacking. We trace this in part to extreme, low-dimensional anisotropy (“rogue dimensions”) and in part to token-length imbalance under mean pooling. A cross-script replication of the rogue-dimension analysis shows this anisotropy is strongly script-invariant on GPT2-small (10/10 dimension overlap between English and Persian backgrounds) but substantially weaker at 7B scale (3–4/10), moderating an initial “globally architectural” hypothesis. We train a lightweight, frozen-backbone projection head with a supervised contrastive objective, a positional-symmetry penalty, rogue-dimension ablation, and a soft PCA-blended initialization, and show it substantially improves orbit recovery on held-out, cross-script, out-of-distribution word forms. On GPT2-small, the full recipe reaches perfect, zero-variance clustering (ARI = 1.000, σ = 0.000 across 5 seeds). On Mistral-7B and Qwen2.5-7B, the same class of intervention, independently re-tuned per model, yields a substantial and bootstrap-significant improvement (ARI = 0.957 ± 0.090 and 0.880 ± 0.115, respectively) though with residual seed variance. Continuous validation directly on embedding geometry confirms that every tested model moves from a hard 0.000 win rate to a hard 1.000 win rate after training, ruling out a discrete-clustering artifact as the source of the reported gains. Critically, since recovering a category label does not guarantee a verified group action, we additionally test the harder, literal claim on a dedicated Aff(Z12) construction with an opaque (non-leaking) position encoding and a shuffled-label control (Appendix F). The retargeted remediation recovers true group position substantially above the shuffled-label control across all three tested architectures (true-label ARI 0.79–1.00 vs. shuffled-control ARI 0.02–0.30), proving the recovery of genuine algebraic structure rather than a correlated category label. We report all of this together with its limitations, an expanded 10x-larger constructed lexicon (Appendix E), natural language generalization tests (Appendix C), a full pre-submission checklist, and complete open-source reproducibility code. Author: Reza Nirouyar ORCID: 0009-0000-4690-6842 Contact: contact@varzin.org Project website: https://varzin.org