Skip to content

Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data

Sep 2026 · arXiv.org · Vol abs/2609.18842 · 1 citation · 104 references
Computer Science

TL;DR

This work proposes the \textbf{Infinite-Parameter LLM}.

Abstract

Scaling laws hold that language models grow more capable with more parameters and more training data. Mixture-of-Experts (MoE) architectures are a remarkable demonstration of these laws, activating only a fraction of an enormous parameter bank for each token. But this success is built on static pretraining data --- the facts and corrections supplied by users during live interactions are a significant untapped source of potential improvement for a deployed model, but cannot be exploited by conventional architectures whose weights are frozen after training. Instead, this newfound knowledge must be placed in the context (by instruction or retrieval) and re-read on every request, only to be discarded afterwards. We seek instead to learn from live interactions by dynamically updating model weights. Inspired by MoEs, we propose the \textbf{Infinite-Parameter LLM}. A compact hypernetwork turns the online data into low-rank modulations of a shared base network, so feed-forward weights are generated from live data, not read from static memory. Whereas existing weight generators are held fixed after reading the context once, we form a Bayesian belief over the generator's latent state and update it online, such that the effective weights are re-derived as our belief evolves during the session. Although the model's memory footprint is constant, the feasible space of generated weights is thus effectively infinite. Representing live data in the weights rather than the prompt amortises compute, frees the context window, persists updates across turns, and can generalise better than in-context use. Our evaluation protocol applies this methodology to in-context learning and retrieval.

View source

Similar papers

Self-Play Pretraining with Zero Data

Self-Play Pretraining with Zero Data is introduced, an initial proof-of-concept towards realizing a more general approach to pretraining, and whether zero-shot performance on natural data improves predictably with self-play compute is tested.

Aditya Cowsik, Kfir Dolev, Michael Y. Li et al. · 3 citations
#machine learning Preprint Oct 2026

Learning to Predict Distributions over Weight Updates for Test-Time Adaptation

Hypernetworks have recently shown success in dynamically adapting the parameters of Large Language Models (LLMs) at runtime based on signals such as task descriptions or additional demostrations. Here we ask: how much adaptation signal can be obtained using only the input query to an LLM?. To answer this, we study quer...

A. Khan, Keshav Ramji, Tahira Naseem et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Where Should a Document Live: Context, Representations, or Parameters?

A controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks shows that in the oracle setting, Cartridges are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points.

Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias et al. · 2 citations
#machine learning Preprint Oct 2026

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

A Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces, which is substantially more token-efficient than forgetting-aw...

Zheng-Hao Zhao, Gao-Wen Liu, Zhi-Ling Lan et al. · 0 citations
#machine learning Preprint Oct 2026

HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoning

Long-form thinking traces can substantially improve the multi-step reasoning performance of large language models (LLMs), but they introduce high inference-time overhead, with latency dominated by sequential decoding. We propose HyperThink, a text-to-parameter approach that amortizes this reasoning computation into a s...

Donggyun Kim, Jack Lu, Chanwoo Kim et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Scalable, Transferable Meta-network for Data Selection Requires a Different Loss (and Why the Obvious Choice is Problematic)

Data selection is critical for training large language models on massive and heterogeneous corpora. Meta-learning for Training-data Selection offers a principled alternative to heuristic scoring by learning data weights from a target validation objective, but existing methods face a trade-off between fine-grained valua...

Zi-Lin Du, Bo-Wen Yang, Bo-Yang Li · 1 citation

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.