Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning
Post-training expert pruning is a promising deployment technique for Mixture-of-Experts (MoE) models. The calibration set is an integral part of the pruning process, however, its correlation with evaluation performance is largely misunderstood and unintuitive. In this work, we present an empirical study of the robustne...
Luis Frentzen Salim, Chia-Chun Wu, Nhiem Tran et al.
· 0 citations