A family of attacks is introduced that eliminates the backward pass by predicting the input gradient from forward-pass hidden states via a lightweight linear regression, and exact affine conditional gradient means are derived.
Kamil Ciosek, Aleksandr V. Petrov, Nicolò Felicioni et al.· arXiv.org· 1 citation
Reliable predictive uncertainty in Laplace approximations depends critically on the prior precision, yet selecting it requires a posterior sweep that is prohibitively expensive for neural networks with billions of parameters. Under the Maximal Update Parametrization ($\mu\mathrm{P}$), we derive a rescaling of the prior...
Richard Bergna, Fernando Ruiz Mazo, Nicolò Felicioni et al.· 0 citations
On a 124M-parameter language model and selected short training windows, the method can lower simulated ledger cost relative to the tested baselines when fleet work is sufficiently cheap, and can lower simulated ledger cost relative to the tested baselines when fleet work is sufficiently cheap.
K. Ciosek, Nicolò Felicioni, Juan Elenter et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.