VCSR-TC: A Sparse Matrix-Matrix Optimization Strategy for Leveraging Modern GPUs with Tensor Cores

Abstract

Sparse–dense matrix multiplication (SpMM) underpins graph neural networks, recommendation systems, and pruned-transformer inference, yet remains memory-bound on modern GPUs because nonzero column indices induce scattered, noncoalesced accesses to the dense operand B. Recent tensor-core SpMM kernels reconstruct the sparse operand into padded dense tiles so it can be fed to Tensor Cores, but pay a per-tile densification cost that grows with sparsity.We introduce VCSR-MM with Stream Tiling, which partitions the column space of A into width-S segments so the corresponding rows of B lie in a bounded slice, then stages that slice in shared memory as a coalesced tile and amortizes the load across every nonzero in the segment. Building on this, VCSR-TC observes that the staged tile produced by Stream Tiling is, by construction, a valid WMMA operand: once S and the output-tile width TK are multiples of the WMMA fragment shape, no runtime, per-invocation densification or per-tile metadata reconstruction is required. Across 200 matrices from the SuiteSparse Matrix Collection, VCSR-TC achieves geomean speedups of 2.40×, 4.44×, and 3.66× over cuSPARSE on NVIDIA A100, H100, and H200 respectively, 1.26×, 2.34×, and 1.93× over DTC-SpMM, and 1.01×, 1.17×, and 1.16× over ASpT. Beyond geomean, VCSR-TC’s performance benefits are more consistent than any other sparse matrix scheme. VCSR-TC’s speedup versus cuSPARSE has a geometric standard deviation ranging 4.11–4.31 across the three GPUs, as compared to ASpT’s (the second best performing scheme) 9.10–10.93, a 2.2–2.5× reduction in cross-matrix spread.