Aug 2026· 1 citation· ⚡ 1 influential· 48 references
Computer Science
TL;DR
MHA-CSP achieves robust structured reasoning via synthetic distance rectification---powered by Mahalanobis-based attention---and efficient information bypass inherited from the CSP backbone, highlighting the effectiveness of complex-valued state propagation with collaborative multi-head rectification in capturing symbolic structures.
Abstract
In this paper, we propose \textbf{Mahalanobis-Based Multi-Head Attention} (MHA-CSP), a novel attention mechanism that replaces the standard dot-product with a \textbf{Mahalanobis distance-based RBF kernel}, which effectively computes attention in an infinite-dimensional feature space without increasing the parameter count. Crucially, the positive definiteness of the Mahalanobis distance enables a \textbf{direct construction of Tree Attention}: attention scores are built directly from accumulated distances, with a LogSumExp correction that rectifies the raw distance by subtracting the log-sum of edge exponentials. Moreover, the multi-head Mahalanobis distance matrices are themselves repurposed to construct an \textbf{attention meshing mechanism}, enabling cross-head kernel collaboration that simultaneously boosts accuracy and training efficiency. Extensive experiments demonstrate that MHA-CSP, with only 119K parameters and \textbf{teacher forcing applied exclusively at the final hidden state}, consistently outperforms Transformer and GCN baselines trained from scratch under identical conditions on long-sequence state tracking tasks. While these baselines rely on dense attention or graph propagation, MHA-CSP achieves robust structured reasoning via synthetic distance rectification---powered by Mahalanobis-based attention---and efficient information bypass inherited from the CSP backbone. This result highlights the effectiveness of complex-valued state propagation with collaborative multi-head rectification in capturing symbolic structures, establishing a new efficiency-performance trade-off for structured reasoning.
Deep neural networks have demonstrated significant potential in direction of arrival (DOA) estimation. However, some existing architectures, especially convolution-based ones, mainly emphasize local feature extraction and may not sufficiently capture long-range dependencies in array observations. To better model such nonlocal correlations, this paper presents a Cascaded Multi-Head Attention Transformer (CMA-Former) for grid-based DOA estimation. The sample covariance matrix is first converted into a compact token sequence using its upper-triangular off-diagonal entries. For each entry, the real part, the imaginary part, and the sine and cosine of its phase are stacked as input features, providing a periodic phase encoding that avoids the discontinuity inherent in raw phase values. A stack of customized Transformer encoders, each equipped with cascaded multi-head attention modules whose head count increases progressively, is then employed to capture sensor-pair correlations across multiple representation subspaces and scales. Finally, a classification token together with a classification head produces confidence scores over a predefined angular grid. Simulation results show that CMA-Former achieves an RMSE lower than or comparable to that of the deep-learning baselines considered. Moreover, it attains a higher estimation success rate for closely spaced sources, indicating an improved capability to resolve adjacent targets. At high SNR, the performance of all grid-based methods is bounded by the off-grid error floor imposed by the fixed angular grid. In addition, a hardware experiment using a cascaded mmWave radar platform further demonstrates the feasibility of applying CMA-Former to real radar measurements without retraining. The source code is publicly available at https://github.com/Syyyt/CMA-Former-official
Yunye Su, Xianpeng Wang, Linqiang Wen et al.· IEEE Transactions on Cogniti...· 0 citations
—Binary Neural Networks (BNNs) are highly attractive for mobile and embedded vision due to their extremely low memory footprint and efficient bit-level convolutions. However, binarization often causes severe information loss and weakens the effectiveness of conventional attention modules designed for full-precision networks. We propose a BNN-oriented attention mechanism, Threshold-Tuning Coordinate Attention (TT-CA), which applies attention by adjusting the binarization decision boundary rather than performing fine-grained multiplicative reweighting. Built upon Coordinate Attention (CA), TT-CA generates a spatially aware threshold ( τ ) from coordinate-wise pooled features and applies a subtractive threshold to induce controllable sign flips, thereby recovering discriminative capability. To balance modulation capacity and deployment efficiency, we quantize the HardSigmoid gating outputs into K-bit discrete levels, enabling lightweight and hardware-friendly gating while avoiding the overly coarse behavior of 1-bit gates. We further explore CA-derived design variants under low-bit settings, including simplified normalization and lightweight bottlenecks, and integrate TT-CA into a compact ResNet14-Wide backbone. Experiments on CIFAR-100 and Tiny ImageNet show consistent accuracy gains over binarized baselines and standard attention plug-ins with modest overhead, reducing Top-1 error by 2.45% on Tiny ImageNet and 0.94% on CIFAR-100. Ablation studies on the threshold strength ( δ ) validate the effectiveness and robustness of threshold tuning for efficient BNNs.
Shaoqing Wu, Hiroyuki Yamauchi· Journal of Advances in Infor...· 0 citations
Recent NVFP4 pretraining work has primarily optimized Transformer linear projections, leaving persistent optimizer states, optimizer computation, and low-precision attention forward--backward paths less explored. We present \textbf{Full-Stack FP4}, a modular NVFP4 framework with separate recipes for projections, AdamW states, Root/Muon computation, and attention. \textbf{LoRA-SVD} protects a compact projection subspace in BF16 while retaining full-shape NVFP4 computation, reducing the linear-only loss gap from \textbf{1.40\%} to \textbf{0.61\%}. An ordered square-root, tile-mean, and Hadamard pipeline enables stable NVFP4 AdamW momentum storage; shape-dependent coefficients and clipping stabilize direct NVFP4 Root iterations; and mixed-precision attention retains softmax-sensitive operations in BF16. On 3B pretraining with 64B tokens, BF16 and Full-Stack FP4 reach losses of \textbf{2.267} and \textbf{2.286}, a \textbf{0.838\%} gap. Their average zero-shot perplexities are 26.675 and 26.665, respectively, with Full-Stack FP4 averaging 0.10 percentage points lower in accuracy. Native four-block measurements on one RTX 5090 show 2.50--2.83$\times$ Root speedups over optimized BF16 and 37.9--42.5\% lower AdamW peak memory.
Siyu Ding, Ming Ma, Jiabo Tong et al.· 2 citations
Background: Multi-head self-attention is central to Transformer-based time-series forecasting, yet its head-level information-selection behavior lacks a unified information-theoretic characterization. How much information a single head selects, how inter-head redundancy should be measured, and under what conditions a head can be removed without degrading predictions remain open questions. Methods: We treat each attention head as a discrete auxiliary selection channel whose conditional distribution is the attention weight vector. This yields a closed-form information identity and an entropy-dependent upper bound on selection information: I(X;Jth)≤logL−E[H(αth)]. We introduce total correlation—the Kullback–Leibler divergence between the joint head distribution and the product of its marginals—as a distributionally principled redundancy measure and relate head-removal sensitivity to conditional task information under population log-loss. Importantly, the selection-information bound characterizes input-dependent positional selection induced by attention weights, rather than the task information carried by the continuous value-weighted head output. Results: Synthetic experiments confirm the entropy-regularized optimality of softmax attention, the selection-information bound, and the redundancy decomposition under controlled conditions. Time-series forecasting experiments across nine benchmark datasets reveal that the head count achieving the lowest observed mean MSE varies across datasets, and that redundancy–sensitivity relationships are dataset- and head-count-dependent, though none remains statistically significant after multiple-comparison correction. Conclusions: The framework provides a principled diagnostic tool for analyzing selection behavior, inter-head dependence, and head-removal sensitivity in multi-head self-attention. It is a diagnostic framework rather than a new forecasting architecture or a standalone pruning algorithm. Pairwise redundancy carries diagnostic signal but is not, by itself, a complete predictor of head-removal sensitivity.
Yan-Bin Zhang, Asif Ahmed Essak, Jiao Ding et al.· Information· 0 citations
Traffic sign recognition is a safety-critical perception task in intelligent transportation systems, requiring accurate classification under complex real-world conditions including illumination variation, viewpoint changes, motion blur, and environmental degradation. Existing methods often rely on single-branch attention mechanisms that capture only partial feature dependencies, limiting robustness under degraded visual conditions. To address these limitations, we propose MCSCA, a Multi-dimensional Collaborative Spatial-Channel Attention network that integrates three complementary attention branches—Neuron Saliency Enhancement (NSE), Spatial-Channel Collaborative Calibration (SCC), and Cross-Dimensional Interaction (CDI)—through a learnable Softmax-weighted adaptive fusion strategy. The three branches operate in parallel on shared intermediate feature maps, simultaneously enhancing neuron-level saliency, spatial-channel contextual dependency, and cross-dimensional structural interaction. The fused representation is further stabilized via residual connection. The proposed model is built upon a lightweight residual backbone with multi-scale feature aggregation and is trained using AdamW with warmup-cosine scheduling, CutMix/Mixup augmentation, and label smoothing. Experiments on GTSRB demonstrate that MCSCA achieves 99.89\% validation accuracy, 99.97\% precision, and 99.80\% recall at 633.4 FPS with only 4.38M parameters, maintaining competitive performance while preserving real-time inference efficiency. Robustness evaluation on GTSRB-C, a corrupted benchmark covering 8 camera corruption types at 5 severity levels, shows a mean corruption accuracy (mCA) of 81.81\% and a composite RobScore of 76.91, with near-perfect robustness under photometric corruptions (Fog mCA: 99.85\%, Rain mCA: 98.87\%) and graceful degradation under additive noise and motion blur. These results validate the effectiveness of the proposed multi-branch collaborative attention design for robust traffic sign recognition under real-world perturbations.
Jinlai Zhang· Poster Volume 0007 The 2026...· 0 citations
The $\mathcal{O}(N^2)$ complexity of attention over $N$ tokens remains a computational bottleneck in transformer models. Vector-Quantized (VQ) attention reduces this to $\mathcal{O}(MN)$ by representing keys with $M$ codewords, but applies uniform codebook capacity regardless of where attention mass concentrates: high-attention regions of key space may be coarsely approximated while low-attention regions waste representational capacity. We propose Adaptive Vector-Quantized (AVQ) Attention, which adaptively allocates codebook capacity based on attention importance. Starting from a small set of codewords, our method identifies the most important codes during the forward pass and refines them with pre-learned child codewords, achieving fine-grained quantization where it matters most while maintaining coarse quantization elsewhere. We develop an implementation using custom Triton kernels that enables the full adaptive refinement process, including importance scoring, child codeword insertion, and parent contribution replacement, to be carried out within the tiled computation paradigm of Flash Attention with minimal overhead. Our approach maintains $\mathcal{O}(MN)$ complexity while achieving improved accuracy-efficiency trade-offs compared to fixed-codebook VQ-attention.
W.V.S.O. van den Dool, Patrick Forré, A. Habibian et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduSep 2, 2026
What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduAug 31, 2026
With millions of users across the world, Julia has been used to conduct cutting-edge research and to design new drugs, jet engines, heat pumps, and more.
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
