Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs
Expert parallelism (EP) enables inference of large Mixture-of-Experts (MoE) models by placing their experts across multiple GPUs, but requires substantial communication between GPUs at every MoE layer. As contemporary MoE models activate more experts per token, this communication accounts for a growing fraction of infe...
Jaehwan Lee, Sang-Min Lee, Chaewon Kim et al.
· 0 citations