Open access

Structure-Derived Bottleneck-Aware Scheduling for Multitasking MCM-GPUs

Tiejian Zhang Guangda Zhang Lu Wang Yandong He Jiping Gao Hengzhu Liu Xia Zhao

Aug 2026 · ACM Transactions on Design Automation of Electronic Systems · 0 citations · 8 references

TL;DR

SA-Scheduler is presented, a structure-derived bottleneck-aware scheduling framework for multitasking MCM-GPUs that determines chip placement without hardware modification or runtime bottleneck profiling, and provides a principled and scalable foundation for multitasking on future MCM-GPUs.

Abstract

Multi-chip-module (MCM) GPUs improve scalability by integrating multiple GPU chiplets, but spatial multitasking introduces severe performance interference due to non-uniform memory access and asymmetric inter-chip bandwidth. Existing GPU multitasking schedulers rely on runtime profiling metrics to guide placement decisions. These reactive approaches observe the consequence of interference rather than its structural cause, vary across inputs and co-running workloads, and incur non-trivial overhead. This paper presents SA-Scheduler, a structure-derived bottleneck-aware scheduling framework for multitasking MCM-GPUs. Instead of profiling execution behavior, SA-Scheduler derives hardware bottleneck regimes directly from kernel structure prior to execution. Through offline architecture-aware modeling, we extract iteration-normalized features such as warp-level working-set footprint and global memory intensity (GMI). A fine-tuned lightweight code model (Qwen-2.5-Coder-1.5B) assists semantic structural extraction for CUDA kernels. These features are translated via an analytical bottleneck model into distinct interference regimes (C, M1, M2, and M3), characterizing how kernels interact with hierarchical and asymmetric memory resources under co-execution. Based on the inferred regimes, SA-Scheduler determines chip placement without hardware modification or runtime bottleneck profiling. For streaming multiprocessor (SM) allocation and optional power-gating decisions, SA-Scheduler further employs lightweight counter-based feedback to tune resource usage under bounded overhead. Across 105 two-program workloads, SA-Scheduler improves system throughput (STP) by 17.63% and 10.48% over intra-chip and inter-chip scheduling, respectively, while increasing fairness by up to 33.27%. An energy-aware Power Mode further reduces average power by up to 25.39% under bounded throughput degradation. These results demonstrate that structure-derived bottleneck inference provides a principled and scalable foundation for multitasking on future MCM-GPUs.

DOI Semantic Scholar

Read PDF

Similar papers

Preprint Jul 2026

Characterizing LLM Kernel Access and Memory Interaction in Multi-Partition NUMA GPUs

Large language model (LLM) workloads motivate multi-partition GPUs as a path to scaling compute and memory capacity, but their non-uniform memory access characteristics and inter-partition communication can amplify contention and degrade locality, leading to suboptimal kernel latency. To address this, we analyze performance-critical LLM kernel implementations spanning weight projection, mixture-of-experts, and attention variants of state-of-the-art serving engines to present a characterization of data access patterns in multi-partition GPUs. First, we introduce memory trace analysis methodology to derive workgroup-level data access and sharing behavior, then evaluate the locality implications on latency using a cycle-level simulator. Using these tools, we categorize LLM kernel operands into three inter-workgroup sharing patterns (global, partial, or private) and show that the required optimization strategies differ across categories, from simple per-workgroup pinning to subgroup-aware co-scheduling. Our findings highlight the need for placement-aware kernel programming and smarter architectural support for work and data locality in multi-partition GPUs.

Donghyeon Joo, Sooraj Puthoor, N. Jayasena et al. · 1 citation

Preprint Jul 2026

CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

This work introduces CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems and demonstrates its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems.

Tingkai Liu, Muralidhar Andoorveedu, Sanjoy Das et al. · 0 citations

Book Open access Jul 2026

DEFT: Joint Task Placement and DVFS for Energy-Efficient Multi-GPU Runtimes

DEFT is presented, an energy-aware scheduling framework that jointly optimizes task-to-device assignment and per-GPU DVFS configuration for task-based multi-GPU applications and employs a cost-model–driven strategy that integrates slack awareness, throughput awareness, and explicit modeling of task execution cost, inter-GPU data movement, and DVFS transition overheads, enabling coordinated placement and frequency decisions under dynamic runtime conditions.

Jing Chen, Miquel Pericàs · International Conference on... · 0 citations

Preprint Jul 2026

GPU-Tile-Sim: A Tile-Centric GPU Simulation Framework for LLM Hardware-Software Co-Design

Modern LLM (large language model) workloads increasingly rely on optimized GPU kernels through hardware-software co-design. These kernels achieve high-performance through fine-grained dependency scheduling and computation-memory overlap. As such, they incur new challenges on existing GPU performance models. Instruction-driven simulators are costly to adapt to evolving architectures, while analytical models are too coarse to capture kernels'characteristics. We propose GPU-Tile-Sim, a tile-centric GPU simulation framework for LLM hardware-software co-design. The key insight is that modern LLM kernel performance is governed less by individual instruction latency than by the dependency structure that controls execution order and overlap. Accordingly, GTSim represents kernel execution as a warp-level tile graph whose nodes capture tile-level operations and whose edges encode data and ordering constraints. Using this representation, we design an automatic tile-graph frontend and a graph-driven simulation backend. We evaluate GTSim on representative GEMM, attention, and end-to-end LLM inference workloads. On A100 and H100 across both conventional and highly optimized kernels, GTSim achieves high performance-modeling accuracy (MAPE, Mean Absolute Percentage Error, 1.22%--8.71%). We further extend GTSim to Blackwell with preliminary validation, and demonstrate its effectiveness in analyzing software and architectural design choices.

Yitong Ding, Jiawei Huang, Renyang Guan et al. · 2 citations · ⚡1

Preprint Aug 2026

Celty: SpMspV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference

Celty, a co-designed sparse format, GPU kernel, and SIMT microarchitecture for efficient spMspV in LLM inference is proposed, which introduces a Run-Length Compressed CSC format that enables vectorized loading of compressed weight columns and exploits both sparsity sources to skip unnecessary memory accesses.

Ruokai Yin, Priyadarshini Panda · 0 citations

Book Open access Jul 2026

Toward Low-Latency and Memory-Efficient Deployment of Irregular Sparse Deep Learning Workloads

This work introduces a tile-aware scheduling framework for efficient sparse Vision Transformer execution on GPUs and introduces a training-aware extension that reuses the inference tile schedule and augments it with backward computation and activation-memory strategies.

Changxin Li · IEEE International Symposium... · 0 citations