We have 4 of 109 papers
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain unclear. We isolate this mechanism by studying normalized SGD in a random-feature model with power-law teacher and data covariance. Fixed-norm...
Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically tractable the model is, and how accurately it describes the true optimization dynamics. In this work, we stress test the simplest possible model...
It is shown that this gating mechanism causes SSMs to first learn an in-weights"memorization"solution, while delaying, or even preventing, convergence to a correct in-context learning solution, and that gating is often beneficial for improving generalization to long sequence lengths.
It is suggested that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
