Skip to content

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Apr 2026 · 0 citations · 44 references
Computer Science

TL;DR

FluxMoE is presented, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new \emph{expert paging} abstraction, and streams weights on demand while keeping expert computation on GPUs.

Abstract

Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inference engines keep all experts GPU-resident, crowding out the key-value cache in large-batch, long-output offline workloads. We present FluxMoE, which decouples experts from physical GPU residency and adapts their footprint to available memory through a new \emph{expert paging} abstraction. FluxMoE combines PagedTensor for transparent remapping, a bandwidth-balanced hierarchy spanning losslessly compressed GPU memory and host DRAM, and a budget-aware residency planner. Unlike CPU-GPU co-inference and whole-layer offloading, FluxMoE streams weights on demand while keeping expert computation on GPUs. We implement FluxMoE atop vLLM and evaluate it on three MoE models. For GLM-4.5 on 8$\times$H20 GPUs, FluxMoE delivers up to 7.2$\times$ vLLM's throughput and 79.0\% lower average Time-Per-Output-Token (TPOT), without measurable model-quality loss using lossless compression. For Mixtral-8$\times$7B-Instruct on 2$\times$L40S GPUs, where weight-resident vLLM cannot fit, FluxMoE delivers 4.3$\times$ KTransformers's throughput and 29.1\% lower average TPOT.

View source

Similar papers

Preprint Oct 2026

MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference

Sparse expert activation reduces MoE models'computation, yet expert weights can exceed limited device memory. Offloading makes inference feasible on a compact AI appliance but exposes host-to-device transfers to the inference path. We present MoE-CORE, a system that coordinates expert offloading and residency for memor...

Ke Yang, Yong-Ji Gao, Xu-Shi Li et al. · 1 citation · ⚡1
Preprint Oct 2026

HiNa-MoE: High-Performance, Non-Intrusive MoE Inference on CPUs with Matrix Engines

Mixture-of-Experts (MoE) inference is increasingly deployed in local and on-premise environments, where expert parameters often exceed GPU memory capacity. In latency-sensitive, low-concurrency settings, repeatedly staging routed-expert weights from CPU memory to the GPU can be prohibitive, leaving routed-expert feed-f...

Wei-Ling Yang, Jun-Wen Zhang, De-Zun Dong et al. · 0 citations
Book Open access Sep 2026

MigMoE: Task-Aware Expert Migration for Faster and More Balanced Expert-Parallel MoE Inference

Motivated by the observation that routing patterns are task-specific, MigMoE is proposed, a task-aware expert migration framework that dynamically adjusts expert placement for multi-task expert parallel MoE inference to balance loads across GPUs.

Xu Han, Zi-Nuo Cai, Zhuo-Long Jiang et al. · 0 citations
Preprint Oct 2026

COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

COPE (Composable Offloading Policy Emulator), which selects the best-performing MoE offloading policy for a target model-platform configuration before runtime integration, and reduces policy-specific implementation effort by up to 74%.

Han-Ming Zhang, Er-Hu Feng · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.