Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs
A growing body of work proposes dynamic expert-skipping methods for Mixture-of-Experts (MoE) models that adjust the number of experts activated per token during inference, aiming to preserve model quality while further reducing the number of active parameters. These methods have added calibrated importance scores, inte...
Jakob Steimle, T. Elsken, Daniel Ehrhardt et al.
· 0 citations