Sparse GEneral Matrix Multiplication (SpGEMM) is one of the most vital kernels in massive research domains, including bioinformatics, graph analytics, and machine learning. Moreover, with the prosperity of the Big Data era, nonzero elements in sparse matrices of SpGEMM boost rapidly into the magnitude of billions. Thus, accelerating billion-scale SpGEMMs becomes the new spotlight. To reduce both the monetary and energy costs of resource-intensive billion-scale SpGEMMs while enhancing their efficiency, we propose CESpGEMM, an adaptive resource-efficient library for billion-scale SpGEMMs on CPU-GPU heterogeneous servers. We design an Anchor-based Sparsity-aware Data Compression (ASDC) mechanism of high compression ratio and throughput to resolve the bottleneck of data transfer through PCIe. Moreover, given an arbitrary SpGEMM, we develop a sampling-based auto-tuning model to determine the optimal hyperparameter configuration. CESpGEMM also involves a hierarchical workflow that exploits adaptive CPU-GPU collaboration. We evaluate the performance of CESpGEMM using one T4 GPU with state-of-the-art SpGEMM libraries. The results demonstrate that compared to the cutting-edge implementation with 4 T4 GPUs, CESpGEMM outperforms these in terms of memory and communication efficiency, and can achieve the average acceleration rate of 5.42×.
Ming Dun, Cheng Zhang, Shuhan Song et al.· IEEE Transactions on Paralle...· 0 citations
Structured sparsity is a promising approach to scaling large-language-model (LLM) inference, but existing forms such as butterfly-structured sparse projections and transformations often map inefficiently to GPUs due to deep stage dependencies and limited bulk parallelism. This paper presents MLX, an algorithm–architecture co-design for structured LLM inference. MLX couples semantic-aware FFT compression and hierarchical sparse projections with spatial dataflow execution, enabling staged structured operators to run efficiently on compact arrays. MLX defines Closed Dependency Components (CDCs) to capture deterministic forward-only dataflow regions that can be folded across layers and pipelined on compact arrays. It then realizes CDCs through a multi-layer execution architecture with bounded-hop skip-hop routing, tag-based scheduling, and decoupled compute/transfer pipelines to overlap communication and computation across deep operators. We prototype MLX in 12 nm and show that it achieves $3.2 \times$ hardware speedup and $3.1 \times$ energy savings over Jetson Xavier. A transformer-specialized reduced design further delivers up to 5.7× speedup over prior sparse accelerators. MLX also scales nearly linearly to $8 \times 8$ meshes and remains effective for long sequences from $\mathbf{1 K}$ to 4 K, demonstrating that structured operator semantics can be translated into efficient spatial execution for sparse LLMs.
Haibin Wu, Wenming Li, Zhihua Fan et al.· International Symposium on C...· 0 citations