Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of e...
Jelena Markovic-Voronov, P. Prastakos, Kayhan Behdin et al.
· 0 citations