Jul 2026· IEEE International Conference on Cloud Computing· pp. 290-296· 0 citations· 30 references
Abstract
The rapid growth of Large Language Models has outpaced GPU memory capacity, making state-of-the-art models unfit for most GPUs. This necessitates evicting model parameters to CPU memory or disk, further away from the processing elements. In these scenarios, inference throughput is restricted by interconnect bandwidth, and while quantization schemes like FP8 reduce memory usage and enable larger models to fit smaller memories, they compromise generation quality and rely on specialized hardware support for efficient inference. To this end, we propose an inference pipeline that utilizes lossless entropy encoding (rANS) alleviating the bottleneck while maintaining standard floating-point portability or output precision. We demonstrate the potential of this approach by integrating the pipeline into vLLM, with a hardware-aware parallelization strategy. Our method increases the effective GPU memory capacity up to 30% in inference workloads with evicted weights, with no loss of quality and at the same time achieving speed up of 2.5–4.4%.
Recent advances in transformer-based foundation models have made them the default choice for many tasks, but their rapidly growing size makes fitting a full model on a single GPU increasingly difficult and their computational cost prohibitive. Block low-rank (BLR) compression techniques address this challenge by learning compact representations of weight matrices. While traditional low-rank (LR) methods often incur sharp accuracy drops, BLR approaches such as Monarch and BLAST can better capture the underlying structure, thus preserving accuracy while reducing computations and memory footprints. In this work, we use roofline analysis to show that, although BLR methods achieve theoretical savings and practical speedups for single-token inference, multi-token inference often becomes memory-bound in practice, increasing latency despite compiler-level optimizations in PyTorch. To address this, we introduce custom Triton kernels with partial fusion and memory layout optimizations for both Monarch and BLAST. On memory-constrained NVIDIA GPUs such as Jetson Orin Nano and A40, our kernels deliver up to 3.76 × speedups and 3 × model size compression over PyTorch dense baselines using CUDA backend and compiler-level optimizations, while supporting various models including Llama-7/1B, GPT2-S, DiT-XL/2, and ViT-B.
Pierre Abillama, Changwoo Lee, Juechu Dong et al.· IEEE International Symposium...· 0 citations
Scaling inference for large language models is increasingly constrained by limited GPU memory, primarily due to the expanding intermediate states (KV caches) required for long-context generation and multi-user workloads. Once the KV cache exceeds the capacity of high-bandwidth memory, it must be offloaded to host memory and reloaded on demand, a workflow severely bottlenecked by the CPU–GPU interconnect, typically PCIe. Existing approaches exploiting offload KV caches to CPU memory and selectively reload partial segments for attention computation often underutilize CPU compute resources and suffer from accuracy degradation. We present Beyond, a drop-in runtime that integrates a smart offloading scheme to selectively identify and retain salient KV entries across continuous decoding sessions, together with a hybrid CPU–GPU attention mechanism for scalable inference. Beyond executes dense attention over recent KV entries stored in GPU memory while performing parallel, per-head sparse attention on salient contextual KV entries residing in CPU memory. The outputs are fused efficiently through a log-sum-exp scheme. During the bandwidth-constrained decoding phase, oversized KV caches are processed cooperatively by the aggregated CPU and GPU memory bandwidth, with only minimal PCIe data movement. Experiments across diverse models and workloads demonstrate that Beyond improves scalability, supports longer sequences and larger batch sizes, and outperforms existing sparse attention baselines in both efficiency and accuracy—all on commodity GPU hardware.
Weishu Deng, Yujie Yang, Peiran Du et al.· IEEE International Symposium...· 1 citation
Prefill or prompt processing and Decode or token generation are two distinct subphases of LLM inference that are greatly influenced by LLM accelerators such as GPT-Generated Unified Format (GGUF Q4_K_M), NormalFloat 4-bit (NF4) Quantization, FlashAttention-2 and others. Although these accelerators clearly improve end-to-end LLM inference performance, their effectiveness over these subphases remains largely understudied. To address this gap, we present a cross-platform, multi-model empirical study, where we deploy multiple ∼ 1B-parameter LLMs on GPU, CPU, and Raspberry Pi 4B edge hardware platforms in the presence and absence of these accelerators. Each test case evaluates 10,000+ inference runs with separate phase-wise and end-to-end performance indicators. Our study brings several important observations, including the contrastive effect of quantization under different hardware bottlenecks, along with a quantification of runtime delays (up to +139%) caused by the lack of parallelism in the ARM architecture. Based on these benchmarking results and observations, we identify several open research challenges in the concluding section. Our work is fully reproducible and open-sourced on GitHub1.
Subhransu Das, Jiaming Cheng, Swathi Vallabhajosyula et al.· Practice and Experience in A...· 0 citations
With the widespread adoption of large language models (LLMs), the demand for graphics processing units (GPUs)-essential for accelerating LLM training and inference- has increased significantly. This has led to rising procurement and operation costs, imposing a significant financial burden on research institutions and industry. Efficient utilization of GPU resources has thus become a critical challenge. In this study, we investigate strategies to maximize resource efficiency by hosting multiple models on a single GPU rather than dedicating each GPU to a single model. We examined various GPU resource partitioning approaches to improve the utilization of limited GPU resources. Specifically, we compared two resource allocation methods for concurrently executing two models on a single GPU: using vLLM, a high-performance LLM inference framework, and using NVIDIA Multi-Instance GPU (MIG). The results demonstrated that the MIG configuration increased total throughput by approximately 700-950 tokens/s compared with vLLM-only execution, suggesting that partitioning a GPU into independent MIG instances can improve throughput for concurrent model execution.
Madoka Takato, Akira Hirai, Reika Kinoshita et al.· Annual International Compute...· 0 citations
The rapid growth of LLM inference has exposed critical bottlenecks in GPU memory capacity and data movement efficiency. When GPU memory is exhausted, modern inference frameworks rely on GPUDirect Storage (GDS) with PCIe P2P transfers to stage data—such as KV caches and intermediate tensors—between GPU and NVMe-based storage. NVMe storage remains a significant performance bottleneck, with state-of-the-art enterprise PCIe Gen5 SSDs delivering only 14 GB/s sequential reads and 7 GB/s writes per drive. In this work, we evaluate a CXL memory expander equipped with an on-device DMA controller as a high-performance staging tier in GDS-style data paths. Although contemporary GPUs do not natively participate in the CXL protocol, our design works around this gap: the device-side DMA controller drives PCIe peer-to-peer (P2P) transfers directly against GPU HBM, while CXL is used only on the host side for capacity expansion and device management. We develop a custom NIXL backend plugin to enable benchmarking of GPU-to-expander data movement, and use NIXLBench to characterize transfer performance across transfer sizes and request parallelism configurations. The expander achieves 51 GB/s read bandwidth (~80% of the PCIe Gen5 ×16 peak) and 33 GB/s write bandwidth—utilizing a substantially larger fraction of the PCIe Gen5 ×16 interconnect than leading Gen5 NVMe SSDs achieve on the same interconnect generation. The remaining ~1.5× read/write gap is consistent with PCIe transaction completion semantics and leaves room for further write-side optimization. Multi-threaded descriptor submission, a routine capability of modern inference frameworks, additionally lowers the block size at which peak bandwidth is reached from 8 MB to 256 KB, sustaining high bandwidth for fine-grained LLM inference workloads such as KV cache updates.
Veerasenareddy Burru, Pradeep Kumar Nalla, Alok Prasad· Proceedings of the ACM SIGCO...· 0 citations
In recent years, Large Language Models (LLMs) have exhibited remarkable capabilities, driving advancements in real-world applications. However, training LLMs on increasingly long input sequences imposes significant challenges due to high GPU memory and computational demands. Existing solutions face two key limitations: (1) memory reduction techniques, such as activation recomputation and CPU offloading, compromise training efficiency; (2) distributed parallelism strategies require excessive GPU resources, limiting the scalability of input sequence length. To address these gaps, we propose Adaptive Sequence Pipeline Parallel Offloading (SPPO), a novel framework that optimizes memory and computational resource efficiency for long-sequence LLM training. SPPO introduces adaptive offloading, leveraging sequence-aware offloading, and two-level activation management to reduce GPU memory consumption without degrading the training efficiency. Additionally, it develops an adaptive pipeline scheduling approach with a heuristic solver and multiplexed sequence partitioning to improve computational resource efficiency. We perform comprehensive experiments to demonstrate the superiority of SPPO: it achieves up to 3.38 × throughput improvement over state-of-the-art methods like Megatron-LM, DeepSpeed and TeraPipe, realizing efficient training of a 7B LLM with sequence lengths of up to 4M tokens on only 128 A100 GPUs.
Qiaoling Chen, Shenggui Li, Wei Gao et al.· International Conference on...· 0 citations