Skip to content
Preprint

TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention

Aug 2026 · 0 citations · 36 references
Computer Science

TL;DR

The work positions the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling, and examines a hierarchical variant of the Neural Cellular Automaton that cascades three stages with windows.

Abstract

Can a strictly local, iterated, weight-shared computation primitive support language modelling, and which of those three properties actually drives the model's behaviour? We define \textsc{TextNCA}, a 1D causal windowed-attention realisation of the Neural Cellular Automaton primitive, and study a hierarchical variant that cascades three stages with windows $w \in \{8, 32, 128\}$ and $T_s$ shared-weight iterations per stage, all on WikiText-103 at roughly 30M parameters and 60k training steps. The model does not match a parameter-matched Transformer at this scale (Hier-TextNCA $60.3$ vs.\ Transformer-6L $52.8$ and Transformer-12L $44.7$ PPL), so we treat it as an analytical probe rather than a proposed alternative. The behaviour we observe is largely explained by the staged narrow-to-wide schedule: a non-iterating sliding-window Transformer that reuses the same schedule comes within $+4.1$ PPL of the iterated model, while reversing, flattening, or breaking the monotonic ordering of the schedule costs between $+16.7$ and $+70.8$ PPL. Iteration adds a smaller bounded benefit on top of the schedule, with a clear optimum at $T_s{=}4$ and a U-shaped degradation beyond it. The GRU gate and learned per-step embeddings are required for that benefit to appear, and training with random $T_s$ yields an inference-time iteration-count knob at the cost of substantially higher absolute PPL. We position the work as a controlled reading of which parts of NCA-style computation carry the weight in language modelling.

View source

Similar papers

Formal Properties of Language as Constraints on Neural Dynamics

It is shown that algebraic properties of language specify invariants that mechanisms must preserve: non-associative hierarchical grouping, commutativity, recursive closure, recursive closure, access to substructures, and structured workspace transitions.

Elliot Murphy · 0 citations
#machine learning Preprint Aug 2026

Fast Weight Attention for Continual Learning

This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models, together with numerically stable positive-decay renormalization, to remain competitive in language modeling and improve length extrapolation on variable-digit addition.

Yi-Fan Zhang, Steve Ta, Jasper Zhang et al. · 0 citations

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

We introduce a simple architectural modification to decoder-only transformers: a persistent recurrent state that observes hidden representations via cross-attention, updates itself through a GRU, and modulates subsequent processing via gated addition. Inserted between the lower and upper halves of a 6-layer transformer...

E. Hering · 0 citations
Preprint Aug 2026

From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does, and a two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does.

Heyang Gong · 0 citations

SMat-Attention: Structured Long-Context Sequence Modeling

This work introduces Structured Matrix Attention via a family of causal masks with structured long-range routing whose row supports have VC-dimension, which makes VC-dimension an explicit knob governing access-pattern complexity, prefill cost, and decoding memory.

E. Anand, Abdullah Ateyeh, Archer Wang et al. · 3 citations

LANTERN: Illuminating Hidden Mathematical Knowledge in Language Models

Language models can now prove theorems, but people still decide which problems to pursue. We ask whether a model's internal representations can help identify promising mathematical connections. We develop LANTERN, a fast, cost-efficient pipeline that uses a classifier over pretrained-model activations to rank candidate...

Pavel Tikhonov, Elena Tutubalina, I. Oseledets et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.