TL;DR
This work proposes Gradient-Coupled Continual Backpropagation (G-CBP), which modulates layer-wise replacement rates from normalized gradient norms and constrains early turnover through an Adaptive Moving Envelope, and identifies an empirical reconstruction–perturbation relationship.
Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify \emph{spectral plasticity collapse}: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leavi...
Dynamical Systems Reconstruction (DSR) aims to infer models from observed time series that reproduce a system's qualitative long-term behavior. Continual DSR (cDSR) requires learning new systems while preserving previously learned dynamics, yet even small parameter updates in recurrent models can qualitatively alter th...
Neural networks trained toward the same final objective can reach similar predictive performance while retaining internal representations shaped by earlier training history. We study this effect using controlled sequential-training experiments in which paired convolutional networks start from identical weights, experie...
Across static, recurrent, attention-based, state-space, continuous-time, implicit, spiking, biologically plausible, and neuromorphic families, forward dynamics have diversified while the highest demonstrated scales remain concentrated in global or closely gradient-derived error-propagation mechanisms.
The learning rate (LR) is the single most influential hyperparameter governing the convergence and stability of stochastic gradient descent (SGD). While numerous heuristic schedules for LR decay and warmup have been proposed to improve generalization, their underlying mathematical behavior as discrete control parameter...
Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict finite-batch stochastic gradient descent (SGD) qualitatively, and we trace the discrepancy to a specific mechanism. In a two-unit ReLU regression, a source task drives the...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
