Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
This work introduces token-error supervision for sparse routing in two forms, one of which directly aligns the router's affinities to the model's objective without requiring an additional head or inference-time modification and both mechanisms preserve the native sparse execution budget and aggregation policy.