Sep 2026· IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems· Vol 45, pp. 4490-4503· 0 citations· 51 references
Abstract
Transformer-based large language models (LLMs) primarily consist of weight-intensive fully connected (FC) layers and cache-dependent attention layers. While batching significantly enhances the throughput of FC layers, it paradoxically increases the cache demands of attention layers. This provides no performance benefit and creates substantial memory pressure. Consequently, existing graphics processing unit (GPU)-based LLM acceleration systems face throughput limitations from batch size constraints. Even when DRAM-based processing-in-memory (PIM) is employed to accelerate attention, the utilization remains extremely low under small batch sizes, which is unsuitable for low-batch scenarios. Fortunately, the emerging nonvolatile resistive random access memory (RRAM) technology offers batch size-insensitive acceleration for FC layers through highly parallel in situ computations by eliminating weight loading overhead. This insight leads us to propose a hybrid approach: RRAM for FC layers and DRAM PIM for attention layers to overcome batch size limitations. However, merely scaling existing RRAM architectures misaligned with LLMs’ computation and storage demands will result in prohibitive overheads. Meanwhile, existing DRAM-based PIMs suffer from poor resource utilization due to the computational pattern of attention layers. Implementing an effective scheduling strategy is equally crucial to harness the potential of the hybrid PIM system. To address these challenges, we present DuoPIM, a novel RRAM–DRAM hybrid PIM architecture optimized for LLM decoding. We introduce novel architectural innovations for both the RRAM and DRAM PIM components to address the challenges posed by LLMs. Specifically, we decouple RRAM’s storage and computing capabilities within a hierarchical architecture, implement minimal modifications to DRAM PIM to support online softmax, and devise dedicated strategies across multiple architectural levels to enhance overall resource utilization. Evaluations demonstrate DuoPIM’s ability to fully leverage computing capacity across various batch sizes.
SLIM (Saturation-Aware Lightweight Performance Model), a semi-analytical model that predicts LLM inference throughput and latency from analytical formulations of Transformer computation and memory traffic, is introduced, which outperforms representative performance-modeling baselines while successfully generalizing to previously unseen operating conditions.
Pol G.Recasens, F. Agulló, Yue Zhu et al.· 0 citations
FLINT is proposed, a workload-driven HBF substrate for capacity-scalable LLM inference that integrates HBF as a memory-capacity tier alongside HBM while addressing three adoption challenges.
Geraldo F. Oliveira, Arash Tavakkol, Xiang-Yu Zhu et al.· 0 citations
Modern large language model (LLM) inference is increasingly dominated by memory-bound operations, making processing-near-memory with processing-in-memory (PNM-PIM) an attractive approach for accelerating the decode phase. However, recent long-context LLMs adopt interleaved local and global attention (ILGA), which introduces heterogeneous execution characteristics across transformer blocks. In realistic serving environments, this heterogeneity is further amplified by mixed-length requests, where inputs with widely different context lengths are processed concurrently. These trends break a key assumption underlying existing PNM-PIM systems that transformer blocks exhibit similar latency and resource demands and can be efficiently mapped to a uniform pipeline. Under ILGA and mixed-length workloads, this assumption no longer holds, leading to severe pipeline imbalance and low utilization in prior PNM-PIM designs. Moreover, PNM-PIM–only systems struggle to efficiently support long-context prefill, which remains compute-intensive and is better suited for GPU execution. In this paper, we propose Aurora, a GPU–PNM–PIM disaggregated system designed to efficiently serve mixed-length LLM inference under ILGA. Aurora introduces an ILGA-aware multi-path PNM-PIM pipeline that explicitly accounts for block-level heterogeneity and request-length diversity, improving pipeline utilization without overprovisioning tensor parallelism. Also, Aurora further adopts a stream-oriented Softmax design to reduce stage-level latency imbalance during decoding. To enable end-to-end inference under disaggregation, Aurora treats KV cache transfer as a pipeline operation and coordinates request issuance and scheduling to avoid transfer-induced stalls. Our experimental results show that Aurora achieves up to 8.5 × and 2.2 × higher throughput than the GPU-only baseline and prior PNM-PIM systems, respectively, in end-to-end Llama4-Scout inference, while requiring a comparable or smaller number of devices.
Hyeonu Kim, Seunghyuk Yu, Minjeoung Kim et al.· International Conference on...· 0 citations
StreamDQ is proposed, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference and reduces latency and improves decode throughput for end-to-end LLM inference.
Minki Jeong, Daegun Yoon, Soohong Ahn et al.· 0 citations
Edge-side LLM deployment, characterized by low-batch inference, has gained significance due to data privacy and personalization requirements. Attention is the key operator of LLM inference, involving massive KV cache access and full-context-aware softmax computation. To optimize KV cache size and softmax computation, various Attention mechanisms and the FlashAttention algorithm have been proposed. As 3D DRAM-based process-near-memory (PNM) presents a promising solution for low-batch LLM inference, processing these optimized Attention mechanisms may suffer from low utilization of 3D DRAM internal bandwidth due to the distributed memory access nature. Furthermore, existing Special Function Unit (SFU) architectures in 3D DRAM PNM for FlashAttention suffer from either severe communication bottlenecks in centralized designs or stringent area constraints in distributed implementations. To address these challenges, we propose H3-Attn, an Attention-efficient 3D DRAM PNM processor for low-batch LLM inference. H3-Attn features a hybrid head parallelism for Attention processing, whereby various optimized Attention mechanisms with spatial tiled FlashAttention can be flexibly enabled with fully leveraged 3D DRAM PNM bandwidth. A Hierarchical SFU (H-SFU) architecture with Multi-Head Instruction Interleaving (MHII) is also proposed, which significantly enhances softmax efficiency in FlashAttention by co-optimizing SFU area footprint and temporal utilization. Evaluations on various LLM models demonstrate that our proposed hybrid head parallelism achieves a 1.54-3.84× decoding speedup over the baseline Attention processing. The H-SFU with MHII delivers a 3.20× compute density improvement compared with prior SFU architectures for 3D DRAM PNM.
Yaolei Li, Wenbin Jia, Zhanchen Zhao et al.· Proceedings of the ACM/IEEE...· 0 citations