Relative Generalization Invariance of LLM Pretraining
Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and training data stream. However, how these components influence performance in distinct ways remains unclear. We take a first step toward isolating their effects by stud...
Feng-Zhuo Zhang, Shu-Che Wang, Sheng-Gui Li et al.
· 0 citations