HEMERA is presented, a heterogeneous memory-centric accelerator for efficient Mamba-2 inference that reformulates the matrix-form SSD computation into an algebraically equivalent streaming-recursive dataflow that avoids quadratic intermediate storage while preserving the original computation.
Abstract
Structured State Space Models (SSMs), such as Mamba, enable efficient long-sequence modeling with linear time complexity. Recent implementations realize this capability through Structured State Space Duality (SSD), which transforms recursive state evolution into matrix-form computations. However, SSD introduces substantial system-level overheads, including quadratic intermediate materialization, irregular data movement, and prefix-dependent execution, leading to excessive memory traffic and bandwidth demand on conventional architectures. Although prior accelerators mitigate these overheads through optimized dataflows or compute-in-memory techniques, they largely retain matrix-oriented SSD execution and cannot simultaneously avoid quadratic intermediate storage and efficiently map dependency-bound state propagation. This paper presents HEMERA, a heterogeneous memory-centric accelerator for efficient Mamba-2 inference. Rather than directly executing the matrix-form SSD computation, HEMERA reformulates it into an algebraically equivalent streaming-recursive dataflow that avoids quadratic intermediate storage while preserving the original computation. The resulting heterogeneous execution paradigm maps dense linear operations onto in-memory computing units and recursive state updates onto a dedicated streaming engine. Across Mamba-2 models ranging from 130M to 2.8B, HEMERA achieves average latency speedups of 1.4x-3.6x and energy-efficiency improvements of 12.2x-27.0x over the official optimized fused Mamba-2 kernel on NVIDIA A100. It further reduces the average SSD-related execution-time ratio across model scales to 14.12% during long-sequence inference, demonstrating its potential for efficient deployment under edge constraints.
Large language model (LLM) inference exhibits substantial variability across adaptation modes, context lengths, and request concurrency, creating challenges for maintaining high utilization, memory efficiency, and scalable performance on compute-in-memory (CIM) accelerators. This paper presents CHIPSMORE, a multi-mode and multi-request LLM inference accelerator that integrates compute-in-interconnect and CIM to support both base-mode and low-rank adaptation (LoRA) inference under diverse workloads. CHIPSMORE employs heterogeneous processing elements consisting of resistive RAM analog compute-in-memory (RRAM-ACIM) and static RAM digital compute-in-memory (SRAM-DCIM) interconnected through a programmable Inter-PE computational network (IPCN). A composable hierarchical key-value (KV) memory scheme dynamically allocates router scratchpad, SRAM-DCIM, and embedded DRAM (eDRAM) resources according to workload requirements, enabling scalable support for long-context and batched inference. Furthermore, a non-replicated multi-request execution pipeline exploits request-level parallelism without duplicating pretrained weights, while a state-aware resource reconfiguration mechanism selectively retains runtime states and power-gates inactive resources to improve energy efficiency. Evaluation using cycle-accurate hardware-software co-simulation demonstrates that CHIPSMORE effectively sustains high throughput across varying model sizes, context lengths, and batch sizes while maintaining favorable power scaling. Compared with Nvidia H100, CHIPSMORE achieves up to $2.38\times$ higher throughput and $27\times$ higher energy efficiency on Mistral-7B inference while eliminating weight replication for multi-request serving.
Matrix-free finite element methods are widely used in large-scale scientific simulations due to their reduced memory footprint and favorable arithmetic intensity, making them particularly attractive for modern GPU architectures. However, achieving high performance for fully fused, matrix-free GPU kernels remains challenging, as execution is often limited by pipeline inefficiencies rather than floating-point throughput. In this work, we present Cheetah, a set of kernel-level optimizations that systematically redesign the execution pipeline of just-in-time compiled matrix-free finite element operators without modifying the mathematical formulation or user-facing programming model. Cheetah improves register reuse, reduces shared-memory traffic and CTA-level (cooperative thread array) synchronization, overlaps indirect memory accesses with computation using asynchronous copy, and exploits constant memory for invariant operator data. These optimizations are implemented transparently within the libCEED JIT framework and require no changes to user-provided quadrature functions. We evaluate Cheetah on mass-like and diffusion-like operators on an NVIDIA A100 GPU. The results demonstrate substantial performance improvements, particularly at polynomial orders p ≥ 4, with up to 30% higher throughput for mass operators and up to 50% for diffusion operators compared to the baseline libCEED GPU backend. Detailed profiling shows that these gains arise from improved compute utilization, reduced synchronization overhead, increased arithmetic intensity, and more effective latency hiding. Our results highlight the importance of execution pipeline design in fully fused matrix-free GPU kernels and demonstrate that low-level kernel restructuring is essential for unlocking the full performance potential of matrix-free methods on modern GPUs.
Jie Ren, H. Ltaief, Stefano Zampini et al.· International Conference on...· 0 citations
FLINT is proposed, a workload-driven HBF substrate for capacity-scalable LLM inference that integrates HBF as a memory-capacity tier alongside HBM while addressing three adoption challenges.
Geraldo F. Oliveira, Arash Tavakkol, Xiang-Yu Zhu et al.· 0 citations
Computing-in-Memory (CIM) accelerators execute Matrix-Vector Multiplications (MVMs) in memory, making them a compelling solution for Machine Learning (ML) workloads. However, existing ML workload partitioning approaches for CIM accelerators do not fully account for Resistive Random Access Memory (RRAM) constraints such as limited memory, high write latency, and limited endurance. They also neglect parallelism, low-level architectural effects, or the Central Processing Unit (CPU) as a complementary compute resource. To address these limitations, we propose an Integer Linear Programming (ILP)-based workload partitioning framework for heterogeneous CPU-CIM systems. It minimizes end-to-end inference latency under RRAM constraints, captures parallelism, and combines empirical profiling with analytical models. Using our framework, heterogeneous CPU-CIM execution achieves speedups of up to 30.9x over CPU-only execution on an edge CPU and 7.3x over a high-performance CPU. A Design Space Exploration (DSE) yields further design insights for future CIM accelerators.
Joel Klein, Rebecca Pelke, Roberto Laudani et al.· 0 citations
Modern edge system-on-chips (SoCs) combine CPUs, integrated GPUs (iGPUs), and neural processing units (NPUs), yet existing LLM runtimes typically make coarse device-level decisions or optimize operators in isolation. As a result, they underutilize heterogeneous resources, particularly on unified-memory platforms where performance depends on both device placement and task-graph coordination. We present HeteroMosaic, a heterogeneity-first scheduling framework for edge LLM inference. HeteroMosaic first uses a heterogeneous roofline model to identify when combining iGPU and NPU execution is beneficial. It then decomposes inference into dependency-preserving micro-batches that expose cross-accelerator overlap and applies trace-guided co-optimization of scheduling and device allocation under practical effects such as memory contention, DVFS, device variation, and NPU runtime overheads. We implement HeteroMosaic in PyTorch C++ and evaluate it on three AMD Ryzen AI platforms spanning NPU-heavy, balanced, and iGPU-heavy designs. On the balanced platform, HeteroMosaic achieves up to 1.73X speedup over an iGPU baseline, 1.78X over an NPU baseline, and 2.05X over frameworks such as llama dot cpp, while reducing energy by up to 45.3%. It also improves performance over prior heterogeneous edge AI solutions by up to 2.35X.
Gregory Jun, Wesley Pang, E. Richter et al.· 2 citations
Embedded systems increasingly require on-device deep learning, yet their processors must simultaneously handle real-time sensing, networking administration, and data control. Existing Field-Programmable Gate Array (FPGA) accelerators typically target peak per-operator throughput without addressing concurrent execution demands of real-time embedded platforms. This paper presents a systolic array-based accelerator prototype implemented on the Zynq-7000 SoC integrated directly into PyTorch, enabling dense linear algebra to be delegated to the FPGA chip while Cortex-A9 continues executing the software stack uninterrupted. Unlike traditional accelerators optimized for peak per-operator speed, this design prioritizes asynchronous co-executionbetween the processing system (PS, the dual-core Cortex-A9) and the programmable logic (PL): The PL performs tiled matrix multiplication, while the PS executes preprocessing, orchestration, and I/O data concurrently, increasing effective end-to-end throughput regardless of the relative isolated performance of CPU and FPGA. The proposed module includes high-level-synthesis (HLS)-based matrix multiplication, activation functions, and Advanced eXtensible Interface (AXI)-Stream Direct Memory Access (DMA) interfaces, wrapped as custom PyTorch kernels under the PetaLinux operating system. The results obtained on the PYNQ-Z2 board show that, once the DMA transfer time is included in the measurement, the FPGA path does not surpass Cortex-A9 in isolated per-operator latencies across the evaluated range; the benefit lies instead in delegating the matrix compute to the fabric at low incremental power while the host CPU cores stay available for concurrent tasks. A concurrent workload sweep across matrix sizes from 8×8 to 256×256 confirms that the co-execution mode sustains 98–99% of available PS compute throughput compared with a constant ≈50% in single-core blocking mode; the difference is statistically significant for all evaluated sizes (see Mann–Whitney U: U=25, p=3.97×10−3, perfect discrimination, n=5). A fair dual-core CPU-only baseline attains comparable PS availability, so this figure reflects the dual-core scheduling that co-execution enables rather than a per-operator advantage of the fabric; the accelerator’s distinct role is to perform the matrix arithmetic off the general-purpose cores at low incremental power. The design occupies only 8% of available look-up tables (LUTs) and 5% of digital signal processing (DSP) blocks, maintains 1.69 W power with a junction temperature of 44.5 °C, and achieves 96.10% MNIST accuracy under fixed-point arithmetic.
Omar Hernandez-Yañez, A. Juárez-Lora, J. Y. Montiel-Pérez et al.· Electronics· 0 citations