A High-Performance and Low-Resource Reconfigurable Hardware Implementation for Softmax in Transformer
Abstract
The efficient implementation of the softmax is critical for optimizing transformer hardware accelerators. Unlike its role as a static, one-time classifier in convolutional neural networks (CNNs), softmax in transformers is core to achieving dynamic contextual awareness, generating attention weights that enable the model to focus on relevant information. Its high-frequency, large-scale execution across every layer makes the softmax a critical performance bottleneck, particularly when processing long sequences. Existing hardware implementations face two primary challenges: excessive resource consumption from the exponential (EXP) computation, and high latency induced by the softmax’s nonelementwise nature. This article proposes a domain-unified piecewise EXP approximation technique. This technique reduces the resource consumption of the EXP unit by 23% and its approximation error by 26%. Furthermore, the proposed design provides direct and efficient hardware-level support for the online softmax algorithm, significantly reducing latency. Under the TSMC 40-nm process, it achieves a $1.36\times $ improvement in area-normalized throughput. On a Xilinx KCU116 FPGA, LUT-normalized throughput improves by $1.26\times $ . Moreover, the proposed reconfigurable architecture substantially improves resource utilization. Finally, end-to-end evaluations confirm that the design strictly preserves native inference accuracy while delivering a notable 4.86% overall system-level latency reduction for Llama3-8B.