Direct-P is introduced for noncausal inference and a causal path that passes the forward quantization directly into backward into forward quantization is addressed, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14$\times$.
Abstract
Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with \emph{Direct-P} for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13$\times$ the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14$\times$. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved backward state can erase the gain. We present \emph{format-aware fusion}, which co-designs each quantization producer with its scale domain and consumer layout for native \...
EFQ-Softmax (Exp-Free Quantization for Softmax), a low-bit probability-generation method that directly maps shifted attention scores to block-scaled E2M1 operands, is proposed and shown to replace the conventional exp-then-quantize path while preserving end-to-end model quality.
Hao-Hui Han, Yu-Ming Wan, Hong-Ning Wang et al.· 0 citations
While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their ex...
Stanislav S. Budzinskiy, Marian Gloser, T. Yilmaz et al.· 0 citations
On Blackwell, normalization and operand movement can limit attention kernels whose matrix multiplications are already deeply pipelined. We implement an FA4-style pipeline in Triton TLX with fixed-shift dense softmax and a saved inverse denominator for backward. The fixed shift removes recurrent accumulator corrections;...
Олександр Сташук, Hong-Tao Yu, J. Shah· 0 citations
Deep Microcompression enables the first documented deployment of a standard CNN on the ATmega328P, a device constrained to 2KB SRAM, previously considered infeasible for CNN inference.
O. M. Busoye, T. M. Busoye, Eghonghon-Aye Eigbe· 1 citation
This work introduces PULSE, a practical codec that enables low-latency decoding on diverse hardware platforms with an ultra-low-complexity 5.2 kMAC/pixel neural receiver, and efficient bit-exact entropy coding with an integer linear CDF predictor and a meta prior.
Zhao-Yang Jia, Tian-Yu Zhang, Zi-Han Zheng et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 14, 2026
Learn how Google DeepMind and GovTech Singapore uses lightweight on-device Gemma models to intercept conversational scams in private chats with zero cloud data egress.
MIT News · Artificial Intelligence· news.mit.eduOct 13, 2026
Machine-learning networks can learn without detailed instructions from a designer. Those that do may offer insight into how the brain reorganizes itself during learning.
