Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their extensive parameter scales pose significant challenges for practical deployment. Unstructured pruning has emerged as an effective model compression strategy with minimal performance loss, which introduces fine-grained sparsity for weight parameters. While existing methods employ a layer-wise pruning strategy to avoid the complexity of global pruning for billion-scale LLMs, they require appropriate sparsity allocation for the layer-wise pruning objectives and often lead to suboptimal solutions for the overall model. In this paper, we propose Lua-LLM ( L earning u nstructured-sparsity a llocation in LLMs), a learning-based global pruning framework that explores the optimal unstructured sparsity allocation. Unlike existing pruning methods, which primarily focus on allocating per-layer sparsity, Lua-LLM achieves flexible allocation for both layer-wise and intra-layer sparsity. Furthermore, Lua-LLM leverages a soft Top-K operator to approximate the importance-based mask selection mechanism, enabling efficient binary mask learning. Experimental results on LLaMA and OPT families demonstrate significant performance improvements over existing methods.
Ming Lu, Jingwei Sun, Junqing Lin et al.· Neural Information Processin...· 1 citation
Low-bit quantization reduces the memory footprint and computational cost of large language model (LLM) inference. However, high-magnitude outlier weights can induce substantial quantization errors and degrade model accuracy. Outlier-aware quantization addresses this issue by retaining outliers in high precision while quantizing the remaining weights, resulting in a low-bit dense GEMM path and a high-precision sparse SpMM path. Existing implementations execute these paths in separate GPU kernels, despite their shared activations and outputs, thereby missing opportunities for intra-operator reuse and incurring redundant global-memory accesses. This inefficiency is particularly pronounced in memory-bound decoding workloads. We propose FlashQuant, a content-sharing execution framework for outlier-aware W4A16 decoding. FlashQuant fuses the dense GEMM and sparse outlier SpMM paths into a single GPU kernel, enabling on-chip reuse of activation and output tiles across heterogeneous computations. It introduces three key techniques: sparse-dense tiling, which aligns outlier processing with dense GEMM tiles; Tile-COO outlier encoding, which enables efficient sparse access and reduces shared-memory bank conflicts; and pipelined scheduling, which overlaps computation with data movement. Experiments show that FlashQuant reduces outlier-processing overhead, achieving $2.74\times - 4.18\times$ speedup over cuBLAS BF16 and up to $1.53\times$ speedup over the strongest unfused outlier-aware baseline.
Junqing Lin, Jingwei Sun, Zhengding Hu et al.· 0 citations
Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates. However, existing decoding implementations often materialize and write back the full recurrent state after every generated token, making state maintenance a major source of memory traffic, especially for models with large states and many heads. This paper presents DeltaLog, a recurrent-state decoding scheme that reduces this overhead without changing the model semantics. Specifically, DeltaLog represents the recurrent state as a dense base state together with a bounded log of recent compact updates. Most decode steps append only compact update factors to this log, while periodic merge steps fold the accumulated updates back into the dense base state. Thus, the model observes the same dense state as in eager decoding, but most full-state write-backs are replaced by lightweight append operations. We implement DeltaLog for GDN, KDA, and RWKV6 and integrate it into a prototype serving stack. Across these models, DeltaLog accelerates the recurrent-state update kernel by up to $1.86\times$, reduces profiled recurrent-state write traffic by up to $7.83\times$, and achieves $1.05$--$1.20\times$ end-to-end serving speedups over dense recurrent baselines.
Depthwise convolution (DWC) is a key operator in efficient convolutional neural networks (CNNs). Recent models increasingly employ large-filter DWC to capture long-range dependencies, achieving accuracy competitive with that of vision transformers. Yet on GPUs, DWC remains far less efficient than its low FLOPs suggest and can become a runtime bottleneck, due to low arithmetic intensity, Tensor-Core mismatch, and inefficient memory access. To tackle these issues, we present Flash-DWC, a GPU library that makes DWC compute-efficient. Flash-DWC introduces a GEMV-based dataflow co-design that jointly optimizes data reuse, access order, layout, and workload tiling. This design increases arithmetic intensity, while preserving access contiguity and coalesced 128-byte memory accesses. Flash-DWC further transforms DWC GEMVs into compact high-density GEMMs on Tensor Cores, enabling efficient mixed-precision acceleration. In addition, matrix compression, a dual-path split-GEMM pipeline, and register-based input reuse further free on-chip resources, hide memory latency, and improve occupancy. We extend these techniques to both forward and backward propagation for efficient end-to-end training. Compared with the fastest cuDNN DWC algorithm, the GEMV-based CUDA-Core kernels and GEMM-based Tensor-Core kernels of Flash-DWC achieve 5.09 × and 9.01 × average speedups, respectively. In CNN training, Flash-DWC attains a 1.03 × –14.18 × speedup and matches PyTorch’s convergence on ImageNet-1K. By making DWC compute-efficient, Flash-DWC promotes the use of larger and more expressive filters.
Zhiyi Zhang, Yang Zhao, Jingwei Sun et al.· ACM Transactions on Architec...· 0 citations
EMO combines graph analysis and pack-level models to formulate energy optimization as a constrained combinatorial problem, efficiently solving for optimal frequency policies under given latency targets.
Jiyu Luo, Shaoyu Chen, Jingwei Sun et al.· 0 citations