We determine how wide a randomly initialized ReLU layer must be to distinguish every pair of inputs. For independent Gaussian weights and zero bias, as the dimensions grow at a fixed output-to-input ratio, the probability of global injectivity tends to zero below a sharp threshold and to one above it. We characterize t...
A Bilevel Local Operator Learning framework for Bayesian inference in PDEs (B-BiLO), which enables efficient gradient-based sampling without synthetic data or adjoint equations and avoids sampling in high-dimensional weight space, as in Bayesian neural networks, by optimizing weights deterministically.
Ray Zirui Zhang, Christopher E. Miles, Xiao-Hui Xie et al.· Nature Communications· 0 citations
Muon replaces the momentum buffer of each weight matrix by its orthogonal polar factor. We ask what this orthogonalization does near an optimum, where minibatch noise dominates the gradient. Under a Gaussian noise model, the expected Muon update becomes a scaled gradient step, so a linear method with a suitably matched...
For a centered, variance-one random variable $X$ with finite fourth moment, let $x$ be the vector of square-free degree-$d$ monomials in $n$ independent copies of $X$. At the critical scale $d^2/n \to \lambda \in (0,\infty)$, the normalized squared length of $x$ converges to the lognormal variable $R = \exp(\sqrt{\lamb...
Xiao-Hui Xie· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.