Xin-Yi Liu

Cobalt: Leveraging Expert Co-activation for Efficient Distributed MoE Training

Mixture-of-Experts (MoE) has increasingly become a mainstream approach for scaling large language models, as it expands model capacity while keeping computation cost nearly constant. Training large-scale MoE models relies on Expert Parallelism (EP), which distributes expert replicas across GPUs and exchanges tokens thr...