SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignment
Mixture-of-Experts (MoE) is a scaling architecture for large language models that activates only a small subset of expert modules per token, enabling massive parameter growth with nearly constant computation. Recent Hybrid MoE architecture adds \textit{shared experts} to capture consistently useful representations, fur...
Qingyu Meng, Yiwei Zha, Jia-Huan Pei et al.
· 0 citations