Jul 2026· International Conference on Supercomputing· pp. 662-674· 0 citations· 44 references
Computer Science
TL;DR
ASTRA is a framework that optimizes the holistic streaming CPD process on GPU that integrates the memory pool and asynchronous strategy with compact storage format to reduce idle cycles and execution latency and devises a similarity-aware updating mechanism to reduce the overall workload.
Abstract
Tensors represent multidimensional data and cover various areas of scientific computing. The Canonical Polyadic Decomposition (CPD) emerges to extract latent patterns from large but highly sparse tensors. In real-world scenarios, tensor slices often arrive dynamically over time in streaming form, making traditional CPD algorithms inefficient in processing the entire tensor at each time step. Streaming CPD processes tensor slices incrementally, exploiting a forgetting factor to adjust the weight of historical information to capture dynamics. Current optimizations mainly focus on CPU platforms, failing to meet the real-time processing requirements of modern applications. Efficiently deploying streaming CPD on GPU remains challenging due to frequent data transfers and memory operations throughout the complex workflow, as well as the intricate computational patterns of bottleneck operators. To address these challenges, we propose ASTRA, a framework that optimizes the holistic streaming CPD process on GPU. At the system level, ASTRA integrates the memory pool and asynchronous strategy with compact storage format to reduce idle cycles and execution latency. At the operator level, ASTRA adopts dedicated techniques to customize the kernel implementations based on operator characteristics as well as an analytical model to adjust the parameters. Finally, ASTRA devises a similarity-aware updating mechanism to reduce the overall workload by skipping the calculations of highly similar adjacent time slices. The experimental results show that ASTRA achieves an average speedup of 1.71 × over the state-of-the-art methods with sufficient accuracy.
Modern GPUs increasingly integrate Tensor Cores into the execution pipeline. Although aggregate tensor throughput continues to grow, aided by an operand supply that has evolved from register-based in Ampere to redundancy-free, memory-based in Hopper and Blackwell, efficiently orchestrating the complete tensor compute pipeline for the modern AI workloads remains challenging. We identify the fundamental bottlenecks as fixed parallelism and coarse-grained scheduling, both of which are exposed by modern AI workloads that interleave diverse non-GEMM operations with GEMM. To orchestrate tensor computation efficiently, we propose FIBER, a new architecture that extends the GPU SIMT (single instruction, multiple thread) model. Its basic execution instance, the \emph{fiber}, is decoupled from private register ownership, carrying only minimal control state while accessing an SM's registers through a shared view. This enables dynamic parallelism scaling, fine-grained register-level dataflow scheduling, and offers a redundancy-free alternative for matrix operand supply. We extend the ISA, microarchitecture, and compiler to realize shared-register addressing, conflict-free operand delivery, and fiber-based program mapping. Under a typical mixed-precision LLM serving scenario, FIBER achieves a 2.25x end-to-end speedup on Ampere (1.15x for the original FP16 computation), with 1.8x and 2.09x on Hopper and Blackwell respectively, and kernel-level gains up to 2.49x.
Zihan Liu, Jingwen Leng, Yangjie Zhou et al.· 0 citations
Matrix completion supports large-scale recommendation and scientific computing, yet existing GPU solvers commonly assume that the observed matrix or its dense factors fit in device memory. On real workloads, this assumption leads to out-of-memory failures or severe PCIe overhead under naive paging. We present TIDE-MC, a bounded-memory GPU framework built on Two-Sided Interpolative Decomposition (TSID). TSID uses a sampled template submatrix as an anchor for reconstructing the full low-rank matrix, allowing computation and storage to scale with the template and active data chunks rather than the complete matrix. TIDE-MC realizes this formulation through two execution stages. First, a conflict-free synchronization engine recovers the template using parallel factorization and hierarchical gradient aggregation. Second, a chunked reconstruction pipeline extends the recovered template to the remaining matrix while overlapping PCIe transfers with GPU computation. An asymmetric gradient-clipping scheme stabilizes mixed-precision Tensor Core execution. Across 15 benchmarks, TIDE-MC completes workloads that cause existing GPU solvers to run out of memory. Compared with the evaluated state-of-the-art baselines, it achieves up to 11,647x speedup, reduces peak memory usage by up to 8.5x, and lowers reconstruction error by up to 99.7%. These results show that template-anchored decomposition and stage-specific GPU execution can scale matrix completion beyond device-memory capacity.
Chengying Huan, Yubo Wang, Pinhuan Wang et al.· 0 citations
Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or expert-level scheduling, which overlooks substantial heterogeneity among tensors within the same layer and adapts poorly to changing hardware load conditions on such devices. This paper presents ATSInfer, a hybrid CPU-GPU inference system for consumer devices that performs offloading at tensor granularity. ATSInfer combines static tensor placement with load-aware dynamic transfer, and introduces asynchronous CPU-GPU coordination to efficiently schedule hardware storage, data movement, and computation across heterogeneous backends. We implement ATSInfer and evaluate it on representative consumer platforms using both dense and MoE models. Compared with existing systems, ATSInfer improves prefill throughput by up to 1.94$\times$ and decode throughput by up to 3.29$\times$, while also increasing GPU utilization and making more effective use of PCIe bandwidth. These results show that ATSInfer can substantially improve the user experience of local LLM deployment on personal consumer devices.
Yangyijian Liu, Hongyi Ye, Mingyang Li et al.· 0 citations
Anticipatory Learning Classifier Systems (ALCS) provide an interpretable symbolic approach to reinforcement learning, but their practical use is often limited by the cost of maintaining evolving rule populations. This paper presents a unified ACS2 framework for comparing five execution regimes: a symbolic baseline, single-process CPU execution, multiprocessing CPU execution, sequential independently seeded tensorized GPU execution, and the same PyTorch tensor backend executed on the host CPU. The contribution is primarily systems- and methodology-oriented rather than algorithmic: the framework enables controlled comparison of runtime, exploit-phase behavior, and maintained population size across distinct ACS2 implementations. The tensorized backend reformulates nearly the full ACS2 learning loop in dense tensor form, including matching, anticipatory updates, offspring handling, and fixed-capacity population management. Benchmarking over the full ACS2 maze suite shows that the multiprocessing CPU backend provides the best overall wall-clock runtime, whereas the single-process CPU backend yields the best average per-experiment runtime. The tensorized modes achieve lower exploit-step averages, but they remain substantially more expensive in runtime terms and must be interpreted cautiously because backend realization also changes population-management semantics. The resulting framework therefore contributes a reproducible basis for studying runtime-population trade-offs and semantic divergence across ACS2 execution strategies.
O. Unold· Proceedings of the Genetic a...· 0 citations
Sparse Matrix-Dense Matrix Multiplication (SpMM) is a dominant computational bottleneck in Graph Neural Network (GNN) inference and training. Representative studies report that SpMM consumes roughly 30% of the execution time in some Graph Convolutional Network (GCN) settings and over 80% in full-batch GraphSAGE training. Despite the rapid growth of GPU SpMM optimization techniques, spanning CUDA core kernels, Tensor Core acceleration, adaptive hybrid execution, autotuning, graph reordering, and framework integration, no dedicated survey has focused on this subfield. This paper presents the first such survey, covering 52 GPU-accelerated SpMM methods for GNN workloads published between 2019 and 2026. We constructed the corpus from IEEE Xplore, the ACM Digital Library, USENIX, arXiv, and Google Scholar, screening the studies first by title and abstract and then by full text. We included GPU-based SpMM kernels and GNN aggregation systems and excluded CPU-only, non-SpMM, abstract-only, and duplicate-version papers. We organize the literature into six technique categories and compare the methods using a ten-dimensional framework. Representative dimensions include sparse format, hardware target, parallelism strategy, load balancing, preprocessing cost, and open-source availability. We find that no single kernel dominates across all graph structures and that effective Tensor Core utilization reaches only 5–20% on irregular GNN matrices. Graph reordering is broadly beneficial, yielding gains of up to $43\times $ when it enables Sparse Tensor Core execution. Because the surveyed literature is overwhelmingly based on CUDA and Tensor Cores, our analysis is NVIDIA-centered. Nevertheless, we distinguish architecture-level insights that generalize to AMD and Intel accelerators from vendor-specific implementation details. We conclude with eight open challenges, including the no-single-winner problem, the Tensor Core utilization gap, standardized benchmarking, and graph-to-kernel compilation.
Tariq Habib Afridi, Muhammad Numan Khan, Young-Koo Lee· IEEE Access· 0 citations
Sparse GEneral Matrix Multiplication (SpGEMM) is one of the most vital kernels in massive research domains, including bioinformatics, graph analytics, and machine learning. Moreover, with the prosperity of the Big Data era, nonzero elements in sparse matrices of SpGEMM boost rapidly into the magnitude of billions. Thus, accelerating billion-scale SpGEMMs becomes the new spotlight. To reduce both the monetary and energy costs of resource-intensive billion-scale SpGEMMs while enhancing their efficiency, we propose CESpGEMM, an adaptive resource-efficient library for billion-scale SpGEMMs on CPU-GPU heterogeneous servers. We design an Anchor-based Sparsity-aware Data Compression (ASDC) mechanism of high compression ratio and throughput to resolve the bottleneck of data transfer through PCIe. Moreover, given an arbitrary SpGEMM, we develop a sampling-based auto-tuning model to determine the optimal hyperparameter configuration. CESpGEMM also involves a hierarchical workflow that exploits adaptive CPU-GPU collaboration. We evaluate the performance of CESpGEMM using one T4 GPU with state-of-the-art SpGEMM libraries. The results demonstrate that compared to the cutting-edge implementation with 4 T4 GPUs, CESpGEMM outperforms these in terms of memory and communication efficiency, and can achieve the average acceleration rate of 5.42×.
Ming Dun, Cheng Zhang, Shuhan Song et al.· IEEE Transactions on Paralle...· 0 citations