The minimax law is proved, showing that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source and giving exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions.
Abstract
How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing $\mu$ for the resulting energy spectrum, we prove the minimax law $\mathfrak R^*_{\rm value}(n,B)\asymp_R \Phi_\mu(n^{-1})+\Phi_\mu(\tau_B), \Phi_\mu(t)=\int\min\{x,t\}\,\mu(\mathrm dx),$ for $n$ prediction blocks and a learned state with at most $2^B$ values. Data set the resolution $1/n$; memory sets the level $\tau_B$ reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy-dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query-key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions. Experiments recover the data-memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.
This work determines the minimax cumulative log-loss regret of a finite-alphabet, exogenously driven source with genuinely infinite input memory, and shows memory decay cannot determine regret, and that a profile-scaled online Newton predictor attains $O_B(\Gamma_T(r)$ in polynomial time per round.
A tractable teacher--student model where a stable latent linear RNN generates trajectories and a sketched linear recurrent student is trained by safeguarded full-batch WSD gradient descent on next-token prediction is studied.
Zi-Yan Chen, Zhong-Zhu Zhou, Pei-Lin Liu et al.· 0 citations
Let $\lambda(n)=(-1)^{\Omega(n)}$ be the Liouville function. We prove a fixed power-logarithmic bound for its logarithmically weighted two-point correlations across the full shift range. There is an absolute $c>0$ such that every sufficiently large $x$ admits a single set $\mathcal E_x\subseteq[1,x]$ with $|\mathcal E_...
A (computationally inefficient) adaptive estimator that, so long as $p$ is a mixture of $k$ symmetric log-concave densities, achieves error comparable with the optimal estimator that knows $p$ and has $\tilde\Theta(n/k)$ samples.
This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models, together with numerically stable positive-decay renormalization, to remain competitive in language modeling and improve length extrapolation on variable-digit addition.
Yi-Fan Zhang, Steve Ta, Jasper Zhang et al.· 0 citations
This work considers the setting where $D$ is a linear differential operator and analyzes a physics-informed kernel estimator combining value observations and differential observations to derive physically consistent rates in a stronger norm that jointly controls the errors in $\hat u$ and $D\hat u$.
Luc Brogat-Motte, Joachim Bona-Pellissier, G. Meanti et al.· 1 citation
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.