Skip to content

TinyCeNN-LM: Quality-Gated Conversion of Pretrained Attention with CeNN-Inspired Cellular-Recurrent Layers

Sep 2026 · 0 citations · 22 references
Computer Science

TL;DR

TinyCeNN-LM introduces a quality-gated post-training conversion framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, fusion, and accept-or-rollback validation to support conservative, quality-gated structural conversion.

Abstract

Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers. TinyCeNN-LM introduces a \emph{quality-gated post-training conversion} framework using CeNN-inspired cellular-recurrent layers with bounded local processing, compact recurrent memory, routing, fusion, and accept-or-rollback validation. Three implementations are studied: Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32. Strict PDelta3 conversion accepts a layer only when representation and NLL criteria pass fixed thresholds. On SmolLM2-135M, layers 0-2 are accepted with cumulative $\Delta\mathrm{NLL}=+0.01209$, while layer 3 is rejected despite acceptable NLL because representation fidelity fails. On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted with final $\Delta\mathrm{NLL}=+0.02073$. Integrated Memory keeps perplexity within $-0.07\%$ to $+0.93\%$ while reducing total cache by up to $6.01\%$. A sampled 200-item downstream sanity check gives $28.5\%$--$32.0\%$ overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion rather than universal attention replacement or speedup.

View source

Similar papers

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

We introduce a simple architectural modification to decoder-only transformers: a persistent recurrent state that observes hidden representations via cross-attention, updates itself through a GRU, and modulates subsequent processing via gated addition. Inserted between the lower and upper halves of a 6-layer transformer...

E. Hering · 0 citations
Conference Open access Sep 2026

Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs

STD is proposed, a hierarchical token pruning framework that adapts token selection mechanisms to the functional role of each network stage, and introduces a Stability-Adaptive Trigger in deep layers to execute pruning only during semantically stable phases.

Shuo Zhang, Jin-Tao Tong, Yi-Xiong Zou et al. · 0 citations
#machine learning Preprint Sep 2026

GTR: Gated Token Recurrence for Efficient Dense Prediction

Results show that recurrent token mixing can provide an efficient alternative to global softmax attention for high-resolution dense prediction and edge deployment, and are shown to be faster than FLA v0.5.

Zhe Feng, Long-Fei Liu, Wei Liu et al. · 0 citations
#machine learning Preprint Sep 2026

UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \nv{} recipe addresses this with current-tensor scaling, a randomized Hadamard transform (RHT), and bfloat16 (BF16) final layers, adding work outside the FP4 matr...

R. Hu, Carlo Luschi, Paul Balanca · 1 citation
#artificial intelligence Preprint Sep 2026

ConvMem: Convolutional Memory for Long-Context Reasoning

ConvMem is a training-free, highly parallelizable framework that reformulates long-context reasoning as a hierarchical convolution, Inspired by CNNs, that outperforms training-free baselines and avoids the risk of overfitting to parametric priors often observed in RL-trained models on out-of-distribution tasks.

Hongming Zhang, Zhao-Zhen Gu, Fengshuo Bai et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.