MIGServe: Layout-Aware Multi-Instance GPU Management for Efficient LLM Serving

Abstract

NVIDIA’s Multi-Instance GPU (MIG) technology partitions a single GPU into hardware-isolated instances of varying sizes, offering a promising substrate for serving Large Language Model (LLM) workloads with dynamic request profiles. However, existing MIG management approaches suffer from high resource fragmentation and reconfiguration overhead. The root cause is that they treat GPU resources as scalar capacities, while MIG enforces a buddy-aligned physical layout in which instances must occupy contiguous slices and can only be reshaped along fixed boundaries. This mismatch raises three challenges: (i) a layout-oblivious small instance acts as a roadblock that prevents adjacent free blocks from coalescing; (ii) stochastic request lifetimes scatter instances across the layout, accumulating fragmentation; and (iii) under skewed traffic, evicting a temporarily idle hot instance for a sporadic cold request triggers an evict-then-reload cycle. We present MIGServe, a layout-aware MIG resource management system that enables fine-grained dynamic reconfiguration for LLM serving. MIGServe treats the physical layout of MIG instances as a first-class scheduling dimension through three techniques: (1) buddy-aware partition placement, which preserves large contiguous free blocks by allocating next to existing occupied buddies; (2) proactive pair-matching migration, which consolidates fragmented half-full buddy pairs off the critical path of inference; and (3) temperature-guided eviction, which shields hot instances from transient cold requests to suppress reconfiguration thrashing. On NVIDIA A100 GPUs with production-inspired LLM workloads, MIGServe serves 2.38 × –5.32 × more requests under 90% SLO attainment, reduces fragmentation by 52.9%–100%, and cuts reconfiguration overhead by 81.7%–100% over state-of-the-art methods.