SMAT: Simple and Efficient Merge-Aware Training
SMAT (Simple MAT) is introduced, which jointly optimizes expert loss and expected loss at simulated merged parameters generated by sampling scaling coefficients, masks, and additive noise, and improves the mean score across five merging methods.
Yang-Gan Gu, Yuan-Yi Wang, Zhen Li et al.
· 0 citations