CARE-MoE: Correlation-Aware Expert Placement and Semantic Equivalence Routing for MoE LLM Inference on Edge Devices

Abstract

Mixture-of-Experts (MoE) has become a mainstream architecture for Large Language Models (LLMs) due to its sparse activation mechanism. While distributed inference is promising for deploying LLMs on resource-constrained edge devices, it still faces two critical challenges for MoEs. First, the gating network’s strong bias toward a small subset of hot experts causes severe cross-device load imbalance. Cloud-side methods, such as expert replication, incur prohibitive memory overhead, and global load balancing introduces excessive communication latency in edge networks. Existing edge methods overlook expert co-activation correlations and cannot adapt to hot-cold expert drift induced by shifting user interactions. Second, the All-to-All communication overhead of expert parallelism can dominate the inference latency under bandwidth-constrained edge environments. Existing token dropping or fusion strategies reduce communication at the cost of distorted token feature propagation and degraded long-context accuracy. To address these challenges, we propose CARE-MoE, an efficient MoE LLM inference framework comprising two core components. The Correlation-Aware Expert Placer (CAEP) balances expert placement by jointly modeling co-activation correlation and hot–cold drift, preventing overload from correlated experts and enabling low-cost adaptive rebalancing. The Communication-Aware Expert Router (CAER) exploits semantic equivalence in MoEs to redirect token-expert assignments toward low-latency devices, thereby achieving significant communication reduction with negligible accuracy loss. Experiments across diverse edge environments and MoE models demonstrate that CARE-MoE achieves a 2.1 × to 5.2 × speedup over state-of-the-art baselines.