Jun 2026· ACM Transactions on Architecture and Code Optimization (TACO)· Vol 23, pp. 1-25· 0 citations· 14 references
TL;DR
Parallax is presented, a fine-grained and accurate performance prediction framework tailored for training–inference co-execution and introduces interpretable modeling strategies for the two primary GPU sharing paradigms.
Abstract
GPU clusters for deep learning (DL) workloads, especially inference, generally suffer from low utilization due to resource over-provisioning for satisfying their strict latency requirements. Co-locating throughput-oriented training jobs with latency-sensitive inference services is a promising approach to reclaim idle resources. However, accurately predicting the performance interference of such heterogeneous workloads under two mainstream GPU sharing mechanisms—Time-Slicing and Multi-Process Service (MPS)—remains a critical challenge. Existing predictors either focus on single-tenant scenarios or lack the fidelity to capture the complex contention patterns between training and inference. In this paper, we present Parallax, a fine-grained and accurate performance prediction framework tailored for training–inference co-execution. Parallax introduces interpretable modeling strategies for the two primary GPU sharing paradigms. For Time-Slicing, we propose a simulation-based model leveraging the virtual time slice and switching overhead abstractions to reconstruct operator-level interleaving and context-switching costs. For MPS, we develop a two-stage framework that first predicts resource utilization under concurrency and then quantifies performance degradation caused by microarchitectural contention, such as memory bandwidth and cache. This resource-centric approach ensures robust generalization to unseen workload combinations. Extensive evaluations on modern GPUs demonstrate the high accuracy of Parallax, predicting execution latency with a MAPE of 4.33% for Time-Slicing and 6.12% for MPS across diverse DL models (averaged over training and inference). Parallax is available at https://github.com/HIT-CeeCG/Parallax.
EPIC mitigates imbalance via performance-aware expert migration and runtime expert activation, and then improves communication with topology-adaptive transport kernels and fine-grained computation-communication overlap.
Jiamin Cao, Qingxu Li, Yaozhong Liu et al.· Proceedings of the ACM SIGCO...· 0 citations
With the rapid advancement of deep learning technology, the parameter scale of large language models has grown exponentially, expanding from hundreds of millions in the early stages to hundreds of billions or even trillions today. Pipeline inference is a crucial approach enabling efficient inference in large language models. However, existing pipeline inference relies on static layer allocation, ignoring the intrinsic variance in layer-wise computation and memory footprints, as well as runtime fluctuations in request rates and sequence lengths. Consequently, under dynamic workloads, compute-dense stages rapidly bottleneck the pipeline and induce severe queue blocking while leaving other devices idle, ultimately degrading end-to-end latency and severe GPU underutilization. To address these challenges, this paper proposes a dynamic pipeline parallel inference algorithm. Centering on the three phases of LLM pipeline inference—partitioning, updating, and migration—the algorithm introduces: (1) A proactive update trigger mechanism driven by multidimensional load forecasting. Rather than relying on reactive bottleneck indicators, it translates projected request rates and token lengths into stage-level VRAM demands, preemptively initiating reconfiguration only when impending hardware capacity violations are detected; (2) A joint partition-migration optimization strategy utilizing a two-stage biased random key genetic algorithm. By embedding a maximum-weight bipartite matching formulation into the evolutionary fitness evaluation, this strategy mathematically couples pipeline boundary search with physical state mapping, maximizing resident parameter reuse to guarantee minimal-overhead model migration; (3) Distributed cluster experiments conducted using public datasets and the Ray framework demonstrate that the proposed method outperforms existing state-of-the-art pipeline inference solutions in metrics including response latency and resource overhead, specifically improving throughput by 2.3% compared to the SOTA framework.
ElastiCo is presented, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms that decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices.
Jinghao Wang, Yihang Zhou, Xiaoyang Sun et al.· 0 citations
With the widespread adoption of large language models (LLMs), the demand for graphics processing units (GPUs)-essential for accelerating LLM training and inference- has increased significantly. This has led to rising procurement and operation costs, imposing a significant financial burden on research institutions and industry. Efficient utilization of GPU resources has thus become a critical challenge. In this study, we investigate strategies to maximize resource efficiency by hosting multiple models on a single GPU rather than dedicating each GPU to a single model. We examined various GPU resource partitioning approaches to improve the utilization of limited GPU resources. Specifically, we compared two resource allocation methods for concurrently executing two models on a single GPU: using vLLM, a high-performance LLM inference framework, and using NVIDIA Multi-Instance GPU (MIG). The results demonstrated that the MIG configuration increased total throughput by approximately 700-950 tokens/s compared with vLLM-only execution, suggesting that partitioning a GPU into independent MIG instances can improve throughput for concurrent model execution.
Madoka Takato, Akira Hirai, Reika Kinoshita et al.· Annual International Compute...· 0 citations
A time-varying integer program to minimize the long-term total cost of the edge AI inference system, including the inference latency, the inference error rate, the query-dispatching communication cost, and the energy consumption, subject to resource and workload constraints is proposed.
Mingtao Ji, Hehan Zhao, Lei Jiao et al.· Science China Information Sc...· 0 citations
Or OrionInfer, an adaptive LLM serving system that aligns inference strategies with real-time demand and introduces three key techniques: runtime switching between data parallelism and tensor parallelism with negligible overhead, an efficient inference pipeline that preserves batching efficiency during parallelism transitions, and live-migration-based load balancing to alleviate memory pressure and improve resource utilization.
Jingqi Feng, Guang Yang, Yukai Huang et al.· Proceedings of the 32nd ACM...· 0 citations