Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Sparse mixture-of-experts (MoE) large language models scale model capacity by routing each token to a small subset of experts. Their routers are regularized with load balancing terms and learn affinity scores through the language-model objective. However, these objectives do not provide direct alignment between routing...
Yury Nahshan, Nati Daniel, Jacob Goldberger et al.
· 0 citations