Skip to content

Kalman Delta Networks: Uncertainty-aware Associative Memory

Sep 2026 · 1 citation · 44 references
Computer Science

TL;DR

This work introduces Kalman Delta Networks (KDNs), a family of linear-attention models that explicitly track memory uncertainty to guide each update, and derives Diagonal KDN and Isotropic KDN, which retain one uncertainty value per key channel and per head, respectively.

Abstract

Linear attention enables efficient long-context inference by compressing token history into a fixed-size recurrent memory. This compression makes each update a trade-off between incorporating new information and preserving useful associations. Models such as DeltaNet, Gated DeltaNet, and KDA predict write strength from the current token representation, without explicitly tracking uncertainty in the stored memory. Yet this uncertainty matters: a new observation should have greater influence when the existing association is uncertain and less when it is already well supported. We introduce Kalman Delta Networks (KDNs), a family of linear-attention models that explicitly track memory uncertainty to guide each update. By formulating associative memory as a linear-Gaussian state-space model, KDNs propagate both the memory estimate and its uncertainty, using the Kalman gain to balance accumulated evidence against the reliability of new observations. This formulation also recovers standard delta-rule updates by replacing tracked covariance with a token-predicted isotropic surrogate. To support hardware-efficient training and inference, we derive Diagonal KDN and Isotropic KDN, which retain one uncertainty value per key channel and per head, respectively. Their uncertainty updates admit associative scans with logarithmic parallel depth, requiring only $O(d_k)$ and $O(1)$ auxiliary state per head. Across controlled pretraining at 750M and 1.3B parameters, both variants consistently improve perplexity and mean downstream accuracy over the evaluated state-of-the-art linear-attention baselines.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

DeltaTTT: Layerwise Optimization for Nonlinear Recurrent Memory

Sequential test-time training adapts a memory network through successive updates, each computing an inner-loop gradient based on the network's previous state. Intuitively, this state dependence should allow each update to account for what the memory has already learned and better incorporate new information. However, w...

Yi-Ning Li, Dong-Chen Han, Jie Fu et al. · 0 citations
#machine learning Preprint Oct 2026

Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory

Recurrent sequence models must decide how strongly to overwrite their memory at each token. Read as Bayesian filtering, this write is the gain of a Kalman update, set by uncertainty from two sources that pull it in opposite directions: volatility, how quickly the underlying associations change, and stochasticity, how n...

Parsa Hejabi, Morteza Dehghani, Payam Piray · 0 citations
#artificial intelligence Preprint Oct 2026

Continuous Memory Machines

Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of...

Ciaran Regan, K. Arulkumaran, L. Darlow et al. · 0 citations
#machine learning Preprint Oct 2026

DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

DeltaNet-PC is developed, which is a PC-native solution for verbatim sequence memory, and can be used for storing single sequences of data, multiple sequences of data, and key-value associative memory pairs with perfect recall accuracy.

Todd Morrill, Richard S. Zemel · 0 citations
Open access Sep 2026

KalWeNN: Kalman-Filtered Memory for Weightless Neural Network Classifiers

Weightless Neural Networks (WNNs) such as WiSARD learn by storing information directly in RAM-based memory structures, enabling fast training and efficient hardware implementation. However, conventional WiSARD memories store only binary activations or occurrence counts, providing no measure of confidence in the stored...

Helena A. B. Moreno, Natasha C. da Fonseca, Gustavo R. Villela et al. · 0 citations

CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

Linear RNNs offer linear-time sequence processing and constant-memory decoding, but their fixed-size recurrent states must accommodate all past key--value associations. Existing forgetting mechanisms and Delta Rule updates reduce interference by selectively clearing or correcting the state, yet earlier associations can...

Yi-Xiao Chen, Shuo-Jin Yang, Shi-Min Hu · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.