PagedMoE, which stores each expert once using a prefix-compatible tree codebook and bitplane layout, and directly executes cached 1-8-bit prefixes without materializing full-precision experts.
Abstract
Large Mixture-of-Experts (MoE) models are promising for local inference because sparse routing allows the routed-expert pool to reside on SSD while DRAM caches recently used experts. However, under practical DRAM cache capacities, substantial SSD I/O remains on the critical path despite I/O-computation overlap. Token-level dynamic expert precision offers a natural way to reduce this data demand by allocating bits according to expert importance and quantization sensitivity. Nevertheless, treating each bit-width of the same expert as an independent object prevents cross-precision cache reuse, causing overlapping data to be fetched repeatedly. The key insight is that precision should instead denote the refinement depth of one shared expert state. Based on this insight, we present PagedMoE, which stores each expert once using a prefix-compatible tree codebook and bitplane layout. Lower-precision representations are valid prefixes of higher-precision ones, allowing cached data to be reused or extended by fetching only missing bitplanes. PagedMoE selects the required prefix length before I/O using current routing weights and offline expert-specific error profiles, and directly executes cached 1-8-bit prefixes without materializing full-precision experts. At an average routed-expert precision of 2.5 bits, PagedMoE achieves quality close to GPTQ-INT4 on Qwen3.5-397B-A17B and to MXFP4 on most evaluated GLM5.2 tasks. With a 24 GiB expert cache, it reaches 8.89 tokens/s on Qwen3.5, 1.33x the throughput of the same dynamic requests using independent precision objects and 1.56x that of uniform 4-bit execution.
Sparse expert activation reduces MoE models'computation, yet expert weights can exceed limited device memory. Offloading makes inference feasible on a compact AI appliance but exposes host-to-device transfers to the inference path. We present MoE-CORE, a system that coordinates expert offloading and residency for memor...
Ke Yang, Yong-Ji Gao, Xu-Shi Li et al.· 1 citation· ⚡1
This work proposes MaskCoFT, a masked co-adaptive fine-tuning method that trains routers and experts together with the cross-entropy loss alone, and reduces expert fetches per token in real offloading system serving.
Junfeng Wu, Ze-Hao Fan, H. Benmeziane et al.· arXiv.org· 0 citations
When a Mixture-of-Experts (MoE) model does not fit in GPU memory, batch-1 decoding is dominated by copying expert weights from host memory. Cache-aware routing reduces these copies by biasing the router toward resident experts, but combined with asynchronous prefetching it makes the model's outputs depend on transfer t...
Mira is proposed, an algorithm-system co-design that enables high-capacity MoE inference on a single GPU by coupling predictive expert management with a tailored quantization format, and reduces expert-induced stalls.
LRMoE is presented, a memory-aware MoE training system that jointly recalibrates backward demand and balances the retained workload, and reduces cumulative MoE-layer training time to fixed-loss targets by 21.2-44.8% relative to SYMI.
Jia-Ming Hu, Zhi-Da Jiang, Z. Xing et al.· 0 citations
The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and...
Yang Zhou, Zhi-Ping Wu, Run-Chang Jiang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.