Skip to content

Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

Sep 2026 · arXiv.org · Vol abs/2609.37751 · 1 citation · 32 references
Computer Science

TL;DR

This work introduces token-error supervision for sparse routing in two forms, one of which directly aligns the router's affinities to the model's objective without requiring an additional head or inference-time modification and both mechanisms preserve the native sparse execution budget and aggregation policy.

Abstract

Sparse mixture-of-experts (MoE) large language models scale model capacity by routing each token to a small subset of experts. Their routers are regularized with load balancing terms and learn affinity scores through the language-model objective. However, these objectives do not provide direct alignment between routing affinities and token-level error. We introduce token-error supervision for sparse routing in two forms. The first form predicts an error score per expert. The affinity-weighted aggregate of these scores is aligned to the next-token cross-entropy loss, while the individual scores attenuate affinity before top-$K$ selection. The second directly aligns the router's affinities to the model's objective without requiring an additional head or inference-time modification. Both formulations use the Itakura--Saito divergence or an exponential negative log-likelihood for aligning affinities and token errors. Across two sparse MoE backbones and four multiple-choice question-answering benchmarks, we evaluate both supervision mechanisms. On Granite, our method improves accuracy by approximately 2.3 percentage points on average over a parameter-matched routing baseline. With stronger supervision, the gain on ARC-Challenge reaches 2.94 points. Both mechanisms preserve the native sparse execution budget and aggregation policy. Our code is available in the supplementary materials.

View source

Similar papers

#natural language process... Preprint Sep 2026

Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts

The Contrastive Routing Mechanism (CoRM) is proposed, which scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state, interpreted through a distinct per-expert projection.

N. Xiros, Dimitrios Damianos, Maria-Eleni Zoumpoulidi et al. · 0 citations
Preprint Aug 2026

Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

This work introduces Giga-Embeddings, a family of text embedding models designed to combine strong retrieval quality with efficient serving, and trains the compact model using a dimension-agnostic objective that aligns teacher and student similarity distributions.

Egor Kolodin, Egor Krasnoperov, Evgeniy Kosarev et al. · 0 citations
#artificial intelligence Open access Sep 2026

Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts

Layer-wise Distribution Alignment (LDA), a lightweight inference-time correction that uses layer-wise calibration statistics to align reduced-routing representations with the default configuration, is proposed, recovering much of the performance lost induced by the distributional shift under reduced routing while prese...

Dohyeon Kim, Bedionita Soro, Sung Ju Hwang · 1 citation
#machine learning Preprint Oct 2026

Principled Top-$k$ Selection for Language Models with Hybrid Gradients

Selecting the best $k$ items out of $m$ candidates is a critical component of modern large language model systems, such as document selection in Retrieval-Augmented Generation (RAG) and expert routing in Mixture-of-Experts (MoEs). However, training these selection modules remains challenging due to weak gradient signal...

Xu-Chen Gong, Ju Sun, Tian Li · 0 citations
Preprint Oct 2026

SheafPrune: Sheaf-Guided Graph Propagation for Visual Token Pruning

Multimodal large language models process the complete visual token sequence, increasing prefill cost even when many tokens are irrelevant to the instruction. Token-wise saliency captures limited inter-token dependence; graph-based selection instead depends on how relations govern importance propagation and ranking. We...

Xiao-Yang Ma, Chi Wang, Hong Sun · 0 citations

Related blog posts

Hugging Face Blog Nov 10, 2026

Decision Models in Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.