Spiking Transformers and Event-Driven Foundation Models: Surrogate Gradient Dynamics, Spike-Driven Attention, and Neuromorphic Silicon Benchmarks

Abstract

Standard digital Transformer architectures rely on dense matrix-matrix multiplications (GEMM) characterized by continuous-valued activations, generating massive thermal energy footprints and memory bandwidth bottlenecks during inference and training. In contrast, biological brains process complex spatio-temporal information via asynchronous, event-driven action potentials (spikes) with extraordinary thermodynamic efficiency. Spiking Neural Networks (SNNs) emulate this biological paradigm by communicating via discrete binary events, replacing energyintensive Multiply-Accumulate (MAC) operations with sparse Accumulate (AC) additions. However, extending spiking architectures to scale-invariant foundation models encounters three critical theoretical and architectural barriers: (1) Non-differentiable threshold-triggered spiking dynamics that prevent canonical backpropagation; (2) Temporal vanishing and exploding gradients across multi-timestep unfolded graphs; and (3) Quadratic complexity self-attention mechanisms that violate binary event sparsity. This paper presents an exhaustive, mathematically formal investigation and empirical benchmark of Spiking Transformers (Spikformers) and event-driven foundation models. We formalize surrogate gradient optimization under parameterized arctangent and fast-sigmoid relaxations, derive linear Spike-Driven Self-Attention (SDSA) operators utilizing pure binary sparse additions, and model asynchronous leaky integrate-and-fire (LIF) dynamics across spatial-temporal horizons. Furthermore, we construct an empirical hardware-in-the-loop neuromorphic testbed benchmarking Spikformers against dense Vision Transformers (ViT) across Intel Loihi 2 neuromorphic processors, SynSense Speck vision sensors, and NVIDIA A100 GPUs. Our findings demonstrate that Spike-Driven Attention reduces computation energy consumption by 82.4% over standard ViT baselines while achieving 81.2% Top-1 accuracy on ImageNet-1K with an average spike firing rate of 0.18 spikes per neuron per timestep. Finally, we formulate open challenges in all-spike autoregressive generation, neuromorphic on-chip continual plasticity, and multi-modal eventsensor fusion.