Skip to content

Relative Generalization Invariance of LLM Pretraining

Sep 2026 · 0 citations · 103 references
Computer Science

TL;DR

This work introduces Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models, and shows that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses.

Abstract

Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by studying relative generalization. We introduce Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models. We show that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately uniform shift in token-wise losses. In contrast, changing the training data stream can substantially alter relative generalization. We further show that RGI cannot be explained by the neural tangent kernel or mean-field regimes alone and prove that it can emerge in an overparameterized quadratic model. Overall, our work identifies RGI as a new phenomenon in LLM pretraining that helps distinguish the effects of optimizers and architectures from those of training data.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed in different ways. Existing work typically evaluates generalization through aggregate accuracy on a single prompt format, task, or set of variations, which conflates robus...

Nagham Omar, Mahmoud Jabarin, M. Rozenshteĭn et al. · 0 citations
Preprint Aug 2026

Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

The empirical results indicate that the reported gains depend heavily on the experiment setup and the choice of random seed, although the trend of stable gains is confirmed with 128-Dyck pretraining of small models with the Llama tokenizer for most of the examined languages.

Sofiia Riazhskykh, Nam Luu, Ondrej Bojar · 0 citations

ProtoSeam: Lifting Classifier Training with Latent Gaussian Mixture Models

Across CIFAR-10, CIFAR-100, and TinyImageNet with ResNet and vision transformer backbones, lifted training improves test accuracy by up to five percentage points over variants without lifting under a shared tuning protocol.

Robert Lampel, Timon Klein, Sebastian Sager · 0 citations
Preprint Aug 2026

From Specialization to Generalization: Instruction-tuned LLMs for Robust Harmful Content Mitigation

By thoroughly unifying 36 English hate speech datasets spanning multiple labeling schemes, this work fine-tune a generalist LLM, based on Qwen3 (Qwen Team, 2025), specifically for hate speech mitigation, demonstrating not only state-of-the-art performance on in-domain benchmarks but also substantial improvements in cro...

Lukas Edman, Daryna Dementieva, Alexander Fraser · 0 citations
#machine learning Preprint Sep 2026

Understanding and Exploiting Anisotropy in Post-Training

LLM post-training combines supervised fine-tuning (SFT), a mode-covering forward-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective to exploit anisotropy, a division of labor that post-training can exploit.

Samyak Jha, Harshvardhan Saini, Yi-Zhen Liao et al. · 0 citations
Preprint Aug 2026

Projector Is All You Train

It is found that training only the projector is sufficient to achieve strong multimodal performance relative to existing baseline models and the authors' jointly trained MLLMs with the same encoder and backbone, and that joint training leads to undesirable drift in existing capabilities of the language model.

Nyx Iskandar, S. Selvan, Slater Victoroff · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.