MoRE: Scaling mixture of experts with hardware-aware low-rank routing
This work proves that rank logarithmic in $M$ suffices for routing expressivity when the number of active experts is fixed, and is necessary up to precision factors, and introduces MoRE (Mixture of Rank-reduced-routed Experts), which factorizes the router weight matrix at rank r and reduces the routing cost to $O((h +...