Enhanced Expert Merging for Mixture-of-Experts in Graph Foundation Models
This work proposes a knowledge distillation-inspired expert merging method that aligns the behavior of parameter-fused experts with expert ensembles, and a theoretical parameter proximity approach that leverages the similarity of expert parameters to approximate ensemble outputs while preserving diversity.