Edge-side LLM deployment, characterized by low-batch inference, has gained significance due to data privacy and personalization requirements. Attention is the key operator of LLM inference, involving massive KV cache access and full-context-aware softmax computation. To optimize KV cache size and softmax computation, various Attention mechanisms and the FlashAttention algorithm have been proposed. As 3D DRAM-based process-near-memory (PNM) presents a promising solution for low-batch LLM inference, processing these optimized Attention mechanisms may suffer from low utilization of 3D DRAM internal bandwidth due to the distributed memory access nature. Furthermore, existing Special Function Unit (SFU) architectures in 3D DRAM PNM for FlashAttention suffer from either severe communication bottlenecks in centralized designs or stringent area constraints in distributed implementations. To address these challenges, we propose H3-Attn, an Attention-efficient 3D DRAM PNM processor for low-batch LLM inference. H3-Attn features a hybrid head parallelism for Attention processing, whereby various optimized Attention mechanisms with spatial tiled FlashAttention can be flexibly enabled with fully leveraged 3D DRAM PNM bandwidth. A Hierarchical SFU (H-SFU) architecture with Multi-Head Instruction Interleaving (MHII) is also proposed, which significantly enhances softmax efficiency in FlashAttention by co-optimizing SFU area footprint and temporal utilization. Evaluations on various LLM models demonstrate that our proposed hybrid head parallelism achieves a 1.54-3.84× decoding speedup over the baseline Attention processing. The H-SFU with MHII delivers a 3.20× compute density improvement compared with prior SFU architectures for 3D DRAM PNM.
Yaolei Li, Wenbin Jia, Zhanchen Zhao et al.· Proceedings of the ACM/IEEE...· 0 citations
Diffusion models have shown marked advancements in 2-D generation and have also become focal points in 3-D generation via consistent multiview image generation. However, the computation and memory demands hinder their real-time deployment on mobile and edge devices. Moreover, the reduction of diffusion timesteps leads to a decrease in interstep similarity, thereby making previous methods ineffective in optimizing computation. The varied layer quantization sensitivity and optimal format in diffusion models also present challenges for traditional quantization methods to achieve efficient memory compression. To address these issues, we first introduce <inline-formula> <tex-math notation="LaTeX">${M}^{3}$ </tex-math></inline-formula> quantization, a mixed-precision, mixed-format, and mixed-granularity quantization framework that allocates optimal precision and format to different data with a unified FP8 computation flow for efficient implementation. Building upon <inline-formula> <tex-math notation="LaTeX">${M}^{3}$ </tex-math></inline-formula> quantization, we present TriM-D, a hardware accelerator designed to optimize computation. It features sparsity-aware dual-branched FP8-MAC units that introduce a dual-branch path and an efficient approximate adder to fully utilize bit-level sparsity of <inline-formula> <tex-math notation="LaTeX">${M}^{3}$ </tex-math></inline-formula>-quantized data. Such an FP8-MAC scheme preserves both efficiency and numerical accuracy. Our experiments demonstrate that TriM-D significantly outperforms the state-of-the-art (SOTA) diffusion accelerator Cambricon-D (Camb-D), achieving an average reduction of 56.5% in memory access and 80.6% in computation cost. In addition, TriM-D provides a <inline-formula> <tex-math notation="LaTeX">$55\times $ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$6.72\times $ </tex-math></inline-formula> improvement in energy efficiency and <inline-formula> <tex-math notation="LaTeX">$3.69\times $ </tex-math></inline-formula>, <inline-formula> <tex-math notation="LaTeX">$2.67\times $ </tex-math></inline-formula> speedups over NVIDIA A100 and Camb-D.
Wenxun Wang, Li-Kai Ma, Chen Tang et al.· IEEE Transactions on Compute...· 0 citations
