Skip to content

Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

Sep 2026 · 0 citations · 55 references
Computer Science

TL;DR

Findings show that routing drift alone is insufficient evidence of routing failure, and source-informed corrections must be judged by their task-level intervention effects, and proposes Selective Router Repair as a case study, and finds that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections.

Abstract

Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such \emph{routing drift} is often interpreted as routing failure, raising a fundamental question that remains unclear: \emph{does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair?} We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as \textit{task loss recoverable under a specified routing intervention, with non-routing parameters fixed.} These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose \emph{Selective Router Repair (SRR)} as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that \textbf{routing drift alone is insufficient evidence of routing failure}: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.

View source

Similar papers

Preprint Aug 2026

RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models

It is found that preserving clean routing substantially recovers steering performance and that routing is more sensitive to semantic content than to behavioral changes under controlled content, which supports routing consistency as an important architectural consideration for adapting representation engineering to MoE...

Zhi-Bo Zhang, Zheng-Mao Ouyang, Ling Shi et al. · 0 citations
Preprint Aug 2026

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

RLCascadeRouter is a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising ``stop''and model selection, and uses trajectory returns and advantages to directly optimize the performance-cost objective.

Shihong Huang, Sheng-Jie Wang, Hong-Yao Ma et al. · 1 citation

A Persistent State for Auditable Mixture-of-Experts Routing

Scratchpad-Augmented Mixture-of-Experts (SA-MoE), which gives each router access to a low-dimensional persistent state that is not provided to the experts, which makes one routing-specific computational history explicit and directly inspectable within the model's natural forward computation.

Abdurrahman Javat, A. Kazakov · 0 citations

FlexRouter: Learning Complementary Model Sets for Flexible LLM Routing

Existing Large Language Model (LLM) routing methods score LLMs independently to select top-$k$ models. However, this ignores model correlations and enforces a rigid computational budget. Consequently, routers often select redundant models that share failure modes, limiting the overall probability of success. To address...

Wang Wei, Harry Yang, Tiankai Yang et al. · 3 citations
#artificial intelligence Preprint Sep 2026

COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

No single Large Language Model (LLM) is uniformly reliable across queries, motivating multi-model inference systems that either route among models or combine their outputs. However, routing stops after selecting an initial model, while dense collaboration invokes peers on every query. We show that collaboration is non-...

Norah Alballa, Wen-Xuan Zhang, Salma Kharrat et al. · 0 citations
#machine learning Preprint Sep 2026

HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion

HERO-MoE, Historical Expert ROuting with Scale-Preserving Fusion, a routing framework that injects historical routing priors into MoE routers by reusing detached, dense routing distributions collected from preceding MoE layers by reusing detached, dense routing distributions collected from preceding MoE layers.

Jun-Xiang Qiu, Zheng-Su Chen, Xin-Ting Hu et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.