Skip to content

One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction

Sep 2026 · 0 citations
Computer Science

TL;DR

The minimax law is proved, showing that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source and giving exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions.

Abstract

How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing $\mu$ for the resulting energy spectrum, we prove the minimax law $\mathfrak R^*_{\rm value}(n,B)\asymp_R \Phi_\mu(n^{-1})+\Phi_\mu(\tau_B), \Phi_\mu(t)=\int\min\{x,t\}\,\mu(\mathrm dx),$ for $n$ prediction blocks and a learned state with at most $2^B$ values. Data set the resolution $1/n$; memory sets the level $\tau_B$ reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy-dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query-key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions. Experiments recover the data-memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.

View source

Similar papers

#machine learning Preprint Aug 2026

Sharp Minimax Regret for Infinite-Memory Logistic Prediction

This work determines the minimax cumulative log-loss regret of a finite-alphabet, exogenously driven source with genuinely infinite input memory, and shows memory decay cannot determine regret, and that a profile-scaled online Newton predictor attains $O_B(\Gamma_T(r)$ in polynomial time per round.

Vaneet Aggarwal · 0 citations
#machine learning Preprint Aug 2026

Linear RNN Scaling Laws: When Longer Sequences Beat More Sequences

A tractable teacher--student model where a stable latent linear RNN generates trajectories and a sketched linear recurrent student is trained by safeguarded full-batch WSD gradient descent on next-token prediction is studied.

Zi-Yan Chen, Zhong-Zhu Zhou, Pei-Lin Liu et al. · 0 citations
Preprint Aug 2026

Logarithmic Chowla Correlations Across All Shift Scales

Let $\lambda(n)=(-1)^{\Omega(n)}$ be the Liouville function. We prove a fixed power-logarithmic bound for its logarithmically weighted two-point correlations across the full shift range. There is an absolute $c>0$ such that every sufficiently large $x$ admits a single set $\mathcal E_x\subseteq[1,x]$ with $|\mathcal E_...

Ji-Zhou Guo · 1 citation · ⚡1
#machine learning Preprint Aug 2026

Fast Weight Attention for Continual Learning

This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models, together with numerically stable positive-decay renormalization, to remain competitive in language modeling and improve length extrapolation on variable-digit addition.

Yi-Fan Zhang, Steve Ta, Jasper Zhang et al. · 0 citations
#machine learning Preprint Sep 2026

Fast Learning Rates for Physics-Informed Kernel Methods

This work considers the setting where $D$ is a linear differential operator and analyzes a physics-informed kernel estimator combining value observations and differential observations to derive physically consistent rates in a stronger norm that jointly controls the errors in $\hat u$ and $D\hat u$.

Luc Brogat-Motte, Joachim Bona-Pellissier, G. Meanti et al. · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.