Higher-order pruning of experts in mixture-of-experts language models
Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count, yet existing methods make pruning decisions for each expert independently, and assume experts'contributions are purely...
Alex M. Tseng, Prannay Kaul, L. Zancato et al.
· 0 citations