Test-time training (TTT) layers replace the linear fast-weight memory of modern recurrent architectures with a small neural network trained online by gradient descent. This nonlinear memory is more expressive than the linear memories of Gated DeltaNet and Mamba-2, but is difficult to parallelize: existing implementatio...
Noel Loo, Neehal Tumma, T. Konstantin Rusch et al.· 1 citation
Linear recurrences are becoming an increasingly common building block in large language models, replacing the growing KV cache of softmax attention with a fixed-size recurrent state that enables efficient decoding. However, the quantization stack has largely been developed around softmax Transformers, whose numerical p...
Philipp Nazari, Neehal Tumma, D. Rus et al.· 1 citation
This work proposes QRR (Quotient Recurrent Reasoner), a gauge-equivariant recurrent architecture with an exact per-token phase-rotation symmetry that achieves stable and faster convergence to solution roots in quotient space, while matching or surpassing recent recurrent reasoning models on Sudoku, Maze, and Mini-ARC.
Flow Language Models (FLMs) have emerged as a continuous-state alternative to discrete diffusion language models, yet the role of their continuous representations in reasoning remains unclear. We investigate this question by comparing the reasoning efficiency of FLMs and discrete diffusion models, measured by solution...
Han-Ru Bai, Faissal Izermine, Oscar Davis et al.· 0 citations
This work proposes a hardware-aware strategy to structurally prune the key and query matrices, reducing the state size while retaining compatibility with existing fast kernels, and proposes a robust structured pruning method based on a rank-revealing QR decomposition.
Looped reasoning models repeatedly apply a shared set of parameters, enabling more computation without increasing the model size. These models also support input-dependent computation by dynamically deciding when to stop looping. Motivated by the recent success of looped transformers in language modeling and reasoning,...
T. Konstantin Rusch, T. Seyde, Jared Boyer et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.