Feedforward spiking neural networks are not transformers (yet): a learning-theoretic framework for long-range dependencies and biological efficiency

Abstract

Spiking neural networks offer a promising route toward low-power sequence computation on neuromorphic hardware, but they continue to lag behind attention-based artificial neural networks on long-context tasks. A central open question is whether this gap reflects only implementation and optimization limitations, or whether architectural features of spiking computation impose unfavorable learnability constraints as sequence length increases. Here, we address this question using a covering-number analysis of feedforward non-leaky integrate-and-fire (nLIF) networks in the probably approximately correct framework. Building on causal-piece decompositions and local Lipschitz continuity, we derive a global sensitivity bound for feedforward nLIF networks and extend it from single-token inputs to multi-token spike sequences. For fixed architectures under stated boundedness and margin assumptions, the resulting sufficient worst-case sample requirement has leading quadratic dependence on sequence length. This dependence arises from cumulative causal participation across time and depth, which increases global sensitivity along active spike paths. We then test the mechanistic implications of this theory using finite-sample cue-recall and teacher–student benchmarks across spiking, recurrent, and attention-based model classes. In cue-recall, an early cue must be retained across distractors and reported at a final query token; in teacher–student, labels are generated by a fixed nLIF teacher, separating representability from finite-sample learnability. Unconstrained feedforward spiking models show sequence-length sensitivity, elevated hidden spike-participation density, and increased samples-to-threshold burden. Post-spike refractoriness, leak-mediated forgetting, learned lateral inhibition, and activity-constrained winner-take-all competition reduce hidden participation and improve empirical robustness in task- and regime-dependent ways. Together, these results identify diffuse causal-set growth as a fundamental architectural bottleneck for baseline feedforward spiking sequence models and suggest that scalable neuromorphic sequence architectures will require circuit mechanisms that explicitly constrain temporal accumulation and effective spike participation.