Skip to content

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Sep 2026 · 0 citations · 35 references
Computer Science

TL;DR

IntBMoE is a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution, and shows consistent gains over representative sparse and dense MoE baselines on image classification.

Abstract

Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.

View source

Similar papers

Preprint Aug 2026

Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

MetaNet is proposed, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias and provides a tunable accuracy-expert-activation trade-off on DeepSeek-MoE-16B-Chat.

Rong-Feng Wang, Shitao Weng, Zhiquan Wang et al. · 0 citations
#machine learning Preprint Sep 2026

RAPTOR: Role-Aware Private Training for Mixture-of-Experts

This work introduces RAPTOR - a Role-Aware Private Training framework, which alternates shared and expert optimization and targets each failure directly, using expert-specific clipping and noise together with a public expected-owner denominator and a count-independent update schedule that avoids conditioning on private...

Duc Dm, Khai Le-Duc, D. Nguyen et al. · 0 citations

RAZOR: Pruning Replaceable Experts in LLMs

RAZOR, a training-free pruning method based on consensus residuals, the deviations of expert outputs from their original weighted mixture, gives the exact output change for a single deletion under survivor renormalization and router refill.

Ming-Yang Song, Mao Zheng · 1 citation
Preprint Aug 2026

UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models

UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem, is introduced, and a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure is introduced.

Lei Xin, Bin Gu, Peize Li et al. · 3 citations

A Persistent State for Auditable Mixture-of-Experts Routing

Scratchpad-Augmented Mixture-of-Experts (SA-MoE), which gives each router access to a low-dimensional persistent state that is not provided to the experts, which makes one routing-specific computational history explicit and directly inspectable within the model's natural forward computation.

Abdurrahman Javat, A. Kazakov · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

MIT News · Artificial Intelligence Oct 23, 2026

Q&A: How AI came to model the “messy” human mind

Assistant Professor Benjamin Lindquist studies how engineers deliberately injected randomness into early AI, giving rise to deep-learning models that sometimes hallucinate.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.