Jul 2026· International Conference on Supercomputing· 0 citations· 34 references
Computer Science
TL;DR
Prism abstracts the highly dynamic diffusion workload into a predictable, static execution flow transparent to the compiler, and achieves this via three techniques: spatial regularization, temporal stabilization, and specialized kernels that selectively bypass padding data.
Abstract
Diffusion-based Large Language Models (DLLMs) promise high-efficiency text generation by refining all tokens in parallel via dense matrix operations. However, this non-autoregressive paradigm introduces a significant challenge: workload dynamism. The number of active tokens fluctuates unpredictably at every denoising step, creating a fundamental conflict with modern deep learning compilers that rely on static tensor shapes for optimization. Consequently, existing serving systems face a dilemma: either incur severe latency spikes due to frequent re-compilation or suffer from low GPU utilization caused by excessive padding. We present Prism, a serving system that reconciles this conflict through a novel paradigm called Virtual Static Execution. Prism abstracts the highly dynamic diffusion workload into a predictable, static execution flow transparent to the compiler. We achieve this via three techniques: (1) Spatial Regularization: We discretize the dynamic token space into a compact set of fixed Physical Buckets, maximizing kernel reuse while minimizing compilation overhead. (2) Temporal Stabilization: We decouple logical diffusion steps from physical execution using a budget-aware scheduler, smoothing out bursty workloads and enabling computation-communication overlap. (3) Compute Optimization: We introduce specialized kernels that selectively bypass padding data, ensuring that the regularization overhead does not translate into wasted compute. Evaluated on LLaDA-8B and Mixture-of-Experts (MoE) models, Prism outperforms state-of-the-art baselines by 5.8 × in throughput and reduces median latency by 70%.
With the rapid advancement of deep learning technology, the parameter scale of large language models has grown exponentially, expanding from hundreds of millions in the early stages to hundreds of billions or even trillions today. Pipeline inference is a crucial approach enabling efficient inference in large language models. However, existing pipeline inference relies on static layer allocation, ignoring the intrinsic variance in layer-wise computation and memory footprints, as well as runtime fluctuations in request rates and sequence lengths. Consequently, under dynamic workloads, compute-dense stages rapidly bottleneck the pipeline and induce severe queue blocking while leaving other devices idle, ultimately degrading end-to-end latency and severe GPU underutilization. To address these challenges, this paper proposes a dynamic pipeline parallel inference algorithm. Centering on the three phases of LLM pipeline inference—partitioning, updating, and migration—the algorithm introduces: (1) A proactive update trigger mechanism driven by multidimensional load forecasting. Rather than relying on reactive bottleneck indicators, it translates projected request rates and token lengths into stage-level VRAM demands, preemptively initiating reconfiguration only when impending hardware capacity violations are detected; (2) A joint partition-migration optimization strategy utilizing a two-stage biased random key genetic algorithm. By embedding a maximum-weight bipartite matching formulation into the evolutionary fitness evaluation, this strategy mathematically couples pipeline boundary search with physical state mapping, maximizing resident parameter reuse to guarantee minimal-overhead model migration; (3) Distributed cluster experiments conducted using public datasets and the Ray framework demonstrate that the proposed method outperforms existing state-of-the-art pipeline inference solutions in metrics including response latency and resource overhead, specifically improving throughput by 2.3% compared to the SOTA framework.
To accelerate large language model (LLM) inference, pipeline parallelism partitions model layers into sequential stages, each assigned to a different device for concurrent execution. However, this method often suffers from pipeline bubbles caused by imbalanced computation in the tail stage. While upstream stages focus solely on layer-forward operations, the final stage must also handle additional post-processing tasks like sampling, which introduces significant latency. This discrepancy in workload leads to pipeline misalignment, forcing upstream stages to idle and degrading overall performance. Existing frameworks typically distribute layers evenly across stages without accounting for computational load differences. To address this, we propose DynaPipe , a dynamic layer redistribution scheme that adaptively balances computation by predicting execution latency in real time. Moreover, we introduce an asynchronous key-value (KV) cache migration coordinator to enable non-blocking layer redistribution during inference. Experiments on representative LLMs demonstrate that DynaPipe reduces average end-to-end request latency by 8% to 41% across diverse workloads, outperforming state-of-the-art pipeline parallelism systems. Our implementation is publicly available at https://github.com/xhx1022/DynaPipe .
Modern large language model (LLM) inference suffers from severe Time-To-First-Token (TTFT) bottlenecks. Existing prefix KV caching mechanisms are inherently stateless, forcing a trade-off between cross-chunk attention accuracy and online recomputation overhead. To address this issue, we propose Pegasus, a novel stateful prefix KV caching system that aims to achieve full-context attention accuracy while avoiding costly recomputation. To handle the exponential growth of context states under limited memory capacity, Pegasus employs a Recursive Path-Pruning Caching (RPPC) algorithm to selectively cache high-value states based on access frequency, memory footprint, and asymmetric latency benefit. In addition, Pegasus introduces a transition-based KV management mechanism to mitigate cache-miss overhead. By exploiting the sparsity of state-dependent KV variations, it replaces expensive attention recomputation and I/O-intensive tensor reloading with lightweight sparse state transitions. Extensive experiments show that Pegasus improves end-to-end serving throughput by 45.9% on average, reduces TTFT by up to 78.5%, and lowers cache-miss recovery overhead by more than 72%.
Fahao Chen, Peng Li, Dongxiao Yu et al.· Fall Joint Computer Conferen...· 0 citations
This work proposes Turbo, a first-of-its-kind in-network aggregation system that accelerates long-context inference by offloading query broadcast and attention aggregation to switches and introduces a rolling forward scheme that propagates states to enable cross-stage updates.
Ying Wan, Yuchen Xu, Chuwen Zhang et al.· Proceedings of the ACM SIGCO...· 0 citations
Or OrionInfer, an adaptive LLM serving system that aligns inference strategies with real-time demand and introduces three key techniques: runtime switching between data parallelism and tensor parallelism with negligible overhead, an efficient inference pipeline that preserves batching efficiency during parallelism transitions, and live-migration-based load balancing to alleviate memory pressure and improve resource utilization.
Jingqi Feng, Guang Yang, Yukai Huang et al.· Proceedings of the 32nd ACM...· 0 citations
DFly is proposed, a block-diffusion framework combining a hybrid target-conditioning backbone with a predecessor-conditioned autoregressive head, improving target-feature utilization and intra-block dependency modeling while keeping generation parallel, and DFly treats verification as a shared batch-level resource.