Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of e...
Jelena Markovic-Voronov, P. Prastakos, Kayhan Behdin et al.· 0 citations
ThinQuant is introduced, a data selection procedure which reduces the required number of calibration data points and an exact reduction of the associated optimization on this reduced calibration set using an efficient ADMM algorithm that iteratively employs thin matrix updates at every step, hence the name ThinQuant.
It is shown that SPLENDID significantly improved prediction accuracy over existing methods, particularly for non-European and admixed ancestries, particularly for non-European and admixed ancestries.
Tony Chen, Hao-Yu Zhang, R. Mazumder et al.· Nature Methods· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.