Experiments show consistent improvements across image classification, event-based recognition, and semantic segmentation andAblation studies and qualitative analyses indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.
Abstract
Spiking Transformers model token interactions primarily through spiking self-attention (SSA). However, binary query and key representations map continuous similarities to sparse and discrete relation responses, which may suppress weak relations and limit the propagation of local spatial context. To address this limitation, we introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF). SLI establishes an attention-independent pathway for direct information exchange among neighboring spiking tokens using lightweight depthwise--pointwise transformations. ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths. The proposed design preserves the original attention formulation and can be incorporated into different Spiking Transformer architectures with modest parameter overhead. Experiments on ImageNet-1K, CIFAR-10, CIFAR-100, CIFAR10-DVS, and ADE20K show consistent improvements across image classification, event-based recognition, and semantic segmentation. In particular, QKFormer with SLI and ACF achieves $84.37\%$ Top-1 accuracy on ImageNet-1K and $37.5\%$ mIoU on ADE20K, where the segmentation model is trained without ImageNet pretraining. Ablation studies and qualitative analyses further indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.
Spiking Transformers provide a promising paradigm for efficient visual processing with spike-driven computation, yet their Softmax-free Spiking Self-Attention (SSA) struggles to establish spatially localized token interactions. Although existing locality-enhanced SSA methods improve accuracy, it remains unclear whether they consistently induce spatial locality across layers and different Spiking Transformer architectures. Through Mean Attention Distance (MAD) analysis, we reveal that computational locality does not necessarily translate into spatial locality and show that uniformly applying the same locality enhancement overlooks architecture-dependent deployment requirements. Motivated by these observations, we propose Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP). SCLA computes attention within non-overlapping regions of spatially adjacent tokens, while BCP facilitates cross-boundary information exchange through a lightweight convolutional pathway. Furthermore, we develop a hierarchical locality deployment strategy to effectively apply SCLA-BCP across the two major Spiking Transformer architectures. Extensive experiments on seven static and neuromorphic datasets covering classification, detection, and segmentation demonstrate consistent improvements with limited parameter and energy overhead. Notably, our approach improves mAP@50 by up to 9.50% on COCO 2017 and mIoU by up to 3.42% on ADE20K. Visualizations, MAD analysis, and ablation studies further validate its effectiveness.
Zeqi Zheng, Zizheng Zhu, Yuping Yan et al.· 0 citations
Spiking Transformers have recently adopted token-wise gating attention mechanisms to avoid the quadratic cost of dot-product attention. However, such token attention mechanisms discard critical relational information between tokens, which limits their expressive power. In this paper, we propose PNT-Attention, a novel attention mechanism designed to address the limitations of existing approaches in Spiking Transformers. PNTAFormer, which incorporates PNT-Attention, enhances token attention by integrating two critical components: (1) 2D local neighborhood aggregation through depthwise convolution to capture spatial context, and (2) excitatory-inhibitory competition driven by membrane polarity in spiking neurons, enabling the model to effectively represent both positive and negative interactions between tokens. Our approach maintains linear complexity and parameter efficiency, while significantly improving the expressive capability of the attention mechanism. Extensive experiments on both static and neuromorphic datasets demonstrate that PNTAFormer achieves significant performance improvements. On CIFAR10, it reaches 96.36% accuracy, outperforming Spikformer by 1.85% (94.51%), and on CIFAR100, it achieves 81.31%, surpassing Spikformer by 3.1% (78.21%). Additionally, PNTAFormer reduces the parameters from 9.32M to 6.74M, maintaining efficiency without sacrificing performance. On CIFAR10-DVS, PNTAFormer further achieves 84.7% top-1 accuracy with 1.5M parameters, demonstrating that the proposed attention mechanism remains highly effective even under strict model capacity constraints and event-based input settings. Moreover, under identical parameter budgets, PNTAFormer consistently outperforms QKFormer across both static and neuromorphic benchmarks, highlighting the importance of restoring token relational information in spike-based attention and demonstrating the effectiveness of polarity-aware local context modeling for Spiking Transformer design.
Lapis is proposed, a spiking attention mechanism that scores each token pair by the L1 distance between its query and key first-spike latency vectors under time-to-first-spike coding, and maps this distance to an affinity through a Laplacian kernel.
Kaiwen Tang, Jiaqi Zheng, Zixuan Zhu et al.· 0 citations
CIS-Fuse is proposed, a spiking network that performs cross-modal fusion directly at the membrane-potential level while inheriting the energy efficiency of spike-based computation, with roughly an order of magnitude lower inference energy than the similarly-sized ANN-based DCEvo.
Rui Zhao, Zhuoyuan Li, Wenrui Li et al.· 0 citations
Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and the mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i)PLMP, a Parallel LIF with Multistage Learnable Parameters neuron and a tailored P-STBP algorithm for stable deep SNN training, (ii) SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax attention with channel-wise spike co-activation and self-compensation, and (iii)SMoE, a spiking mixture-of-experts module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines. Under a standard MAC/AC arithmetic model, SMSA reduces the estimated operator-level compute energy of the attention module by up to 97%, while whole-model profiling shows more moderate but consistent efficiency gains.
Xiubo Liang, Jinxing Han, Yuke Li et al.· 0 citations
Spiking neural networks (SNNs) have garnered significant attention in reinforcement learning tasks for their low power consumption. However, traditional spiking reinforcement learning (SRL) methods, which rely on local-connected encoding and fixed-threshold learning, struggle to capture the inter-dimensional correlations of input information within short timesteps, limiting the network’s expressive capacity at low timesteps. While increasing timesteps can significantly enhance performance, excessive timesteps result in substantial delays. To address this contradiction and enhance the expressive and decision-making capabilities of SNNs within short timesteps, we propose Mask-Adaptive Global Connection (MAGC), a novel encoding method that efficiently captures long-range dependencies via sparse, adaptively masked connections—enabling global feature interaction in a single timestep. Additionally, dynamic-threshold spiking neurons are introduced to effectively capture and distinguish subtle changes in input signals at each timestep, thereby enhancing the spatial-temporal state representation during spike information transmission. Extensive experimental results demonstrate that the proposed method achieves performance comparable to state-of-the-art algorithms using only a single timestep, while significantly reducing inference latency and energy consumption. When extended to multiple timesteps, our approach consistently outperforms existing methods, showing substantial improvements across eight continuous control tasks from OpenAI Gym.
Rong Xiao, Zhiyuan Hu, Ping He et al.· IEEE Transactions on Image P...· 0 citations