Aug 2026· Proceedings of the ACM SIGCOMM 2026 Conference· 0 citations· 33 references
TL;DR
EPIC mitigates imbalance via performance-aware expert migration and runtime expert activation, and then improves communication with topology-adaptive transport kernels and fine-grained computation-communication overlap.
Abstract
The Mixture-of-Experts (MoE) architecture scales large language models (LLMs) to trillions of parameters by activating only a small subset of experts per token. In practice, MoE inference is commonly deployed with Expert Parallelism (EP), which places whole experts on different GPUs to preserve kernel efficiency. However, production EP deployments often suffer from two bottlenecks: (1) expert workload imbalance, which creates computation and communication stragglers, and (2) communication inefficiency, where inter-GPU transfers dominate latency even after balancing. We present EPIC, an experience-driven EP inference system that addresses these issues progressively for real deployments. EPIC mitigates imbalance via performance-aware expert migration and runtime expert activation, and then improves communication with topology-adaptive transport kernels and fine-grained computation-communication overlap. EPIC has been deployed at scale across O(10K) GPUs in our online inference service for both open-source models (e.g., Qwen3-Coder and DeepSeek-R1) and internal models, reducing communication time and per-token latency by up to 40% and 21%, respectively.
This work presents a fine-grained approach that overlaps expert compute with the second all-to-all via tile-level signaling and scheduling, making it practical for improving distributed MoE execution efficiency on multi-GPU systems.
Minyu Cui, Anna Wingkvist, Morgan Ericsson· 1 citation
Or OrionInfer, an adaptive LLM serving system that aligns inference strategies with real-time demand and introduces three key techniques: runtime switching between data parallelism and tensor parallelism with negligible overhead, an efficient inference pipeline that preserves batching efficiency during parallelism transitions, and live-migration-based load balancing to alleviate memory pressure and improve resource utilization.
Jingqi Feng, Guang Yang, Yukai Huang et al.· Proceedings of the 32nd ACM...· 0 citations
MaverIQ is an intent-based LLM inference serving system that automatically maps user intents to deployment configurations while minimizing operational cost for the provider and reduces profiling overheads by 7-15× compared to state-of-the-art baselines.
Dimitrios Liakopoulos, Prasoon Sinha, Tianrui Hu et al.· ACM SIGOPS Operating Systems...· 0 citations
With a trace-driven, event-atomic simulator over three MoE models, a large offline-optimal gap substantially overstates the gains recovered by representative lightweight causal mechanisms.
Large language models have become increasingly capable agents for low-level code and kernel optimization, but isolated kernel benchmarks provide only a proxy for the deployment behavior that matters in language-model inference. We identify a benchmark-to-deployment gap: candidate kernels that appear correct and fast in standalone harnesses can exhibit different performance, safety, or phase behavior after integration into a real inference workload. We introduce LLM4LLM, a deployment-aware closed-loop optimization framework that starts from a target inference script, extracts phase-aware optimization tasks, searches with an experience-guided episodic agent, and accepts patches through in-model validation. Across ten language-model inference workloads on A100 and H100 GPUs, LLM4LLM improves end-to-end latency for every evaluated model, achieving 3.91$\times$/6.98$\times$ geometric-mean speedups on A100/H100; as supporting kernel-level evidence, it also attains up to 2.745$\times$ GeoMean speedup on KernelBench Level 2.
Hui Zeng, Pengfei Yang, Yanxin Chen et al.· 0 citations
With the rapid advancement of deep learning technology, the parameter scale of large language models has grown exponentially, expanding from hundreds of millions in the early stages to hundreds of billions or even trillions today. Pipeline inference is a crucial approach enabling efficient inference in large language models. However, existing pipeline inference relies on static layer allocation, ignoring the intrinsic variance in layer-wise computation and memory footprints, as well as runtime fluctuations in request rates and sequence lengths. Consequently, under dynamic workloads, compute-dense stages rapidly bottleneck the pipeline and induce severe queue blocking while leaving other devices idle, ultimately degrading end-to-end latency and severe GPU underutilization. To address these challenges, this paper proposes a dynamic pipeline parallel inference algorithm. Centering on the three phases of LLM pipeline inference—partitioning, updating, and migration—the algorithm introduces: (1) A proactive update trigger mechanism driven by multidimensional load forecasting. Rather than relying on reactive bottleneck indicators, it translates projected request rates and token lengths into stage-level VRAM demands, preemptively initiating reconfiguration only when impending hardware capacity violations are detected; (2) A joint partition-migration optimization strategy utilizing a two-stage biased random key genetic algorithm. By embedding a maximum-weight bipartite matching formulation into the evolutionary fitness evaluation, this strategy mathematically couples pipeline boundary search with physical state mapping, maximizing resident parameter reuse to guarantee minimal-overhead model migration; (3) Distributed cluster experiments conducted using public datasets and the Ray framework demonstrate that the proposed method outperforms existing state-of-the-art pipeline inference solutions in metrics including response latency and resource overhead, specifically improving throughput by 2.3% compared to the SOTA framework.
