Control Parameter Dynamics in Neural Network Optimization: An Analysis of Learning Rate Strategies for Sequential Data Modeling

Abstract

The learning rate (LR) is the single most influential hyperparameter governing the convergence and stability of stochastic gradient descent (SGD). While numerous heuristic schedules for LR decay and warmup have been proposed to improve generalization, their underlying mathematical behavior as discrete control parameters remains insufficiently characterized. In this paper, we consider the SGD optimization process as a first-order discrete dynamical system, where the learning rate acts as a time-varying gain coefficient. We present a comprehensive mathematical analysis of ten distinct LR schedules-comprising four decay strategies (Linear, Exponential, Cosine, and Sigmoid), three warmup strategies (Piecewise Constant, Linear, and Exponential) and three mixed strategies (Staircase, Trapezoid and Sigmoid, plus Warmup)—deriving their closed-form iterative expressions, contraction properties, and asymptotic convergence rates. This analytical framework provides a rigorous foundation for understanding why specific schedules succeed or fail in non-convex optimization landscapes. Subsequently, this work lays the groundwork for empirical validation using recurrent architectures on time-series forecasting tasks.