Results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics, and Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce.
Abstract
We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.
The results suggest that continual learning is not a single capability: different patterns of environmental change require fundamentally different update behaviors, determining when adaptation must be learned inside model weights and when it can be achieved through external scaffolding.
A. Harrington, Nayan Saxena, Michael Murphy et al.· 1 citation· ⚡1
We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates and batch sizes, we investigate their marginal evolution with model capacity and data scale and develop a model that captures these relationships. As we employ a Warmup-Stable-Decay learning rate schedule, we further investigate the gains from learning rate annealing over a broad range of hyperparameters settings, models and data budgets, and whether the optimal learning rate and batch size transfer between the stable and decay phases. Finally, we characterize the dependence of loss on model capacity and dataset size, evaluating recently proposed scaling forms that explicitly model their interaction. We find these approaches particularly effective at capturing both undertraining and overtraining regimes across our experiments. This study establishes a first baseline and scaling procedure for the development of future OpenEuroLLM models. We open-source the complete collection of pretraining runs used in this study.
Niccolò Ajroldi, Diana-Alexandra Onutu, Haider Al-Tahan et al.· 0 citations
Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard approach for improving complex reasoning in Large Language Models (LLMs). However, entropy collapse remains a critical bottleneck, often leading to performance degradation. While typically attributed to reduced exploration, we identify a more severe failure mode: update deadlock. We show that policy entropy bounds both the policy gradient and probability update norms; consequently, entropy collapse effectively stops reward signal backpropagation, preventing further policy learning regardless of data quality. To analyze this, we develop a micro-dynamics framework based on single-step logit updates. Our results indicate that entropy evolution depends on the interaction between advantage signals and the log-probability residual relative to its expectation, offering a unified perspective on existing interventions. Based on these findings, we propose ANCHOR, an algorithm designed to resolve deadlocks and maintain stability. ANCHOR uses a dual mechanism: it dynamically anchors the entropy decay of positive samples to prevent premature saturation, and uses entropy variations to stabilize updates for negative samples. Extensive experiments on reasoning benchmarks show that ANCHOR significantly outperforms strong baselines. Unlike heuristic methods that induce verbosity to mitigate collapse, ANCHOR promotes concise reasoning paths, improving both stability and efficiency.
Cong Qin, Jiaye Lin, Xiaoliang Fu et al.· Proceedings of the 32nd ACM...· 0 citations
A systematic literature review on how RL are adapted and scaled as a fundamental post-training tools and how innovations in the RL pipeline enhance the domain-specific LLMs is conducted.
Qianyue Hao, Lin Chen, Xiaoqian Qi et al.· ACM Computing Surveys· 1 citation
The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model ($\text{PPL}<20$) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released$^{\S}$.
Behavior manifold analysis is introduced, which isolates behavior-specific geometry by selecting sparse behavior-associated coordinates and lifting them into low-dimensional local charts, and provides a unified framework for understanding the mechanistic distinction between the two objectives.
Juntong Wang, Shengkun Yang, Xiyuan Wang et al.· 0 citations