Multi-GPU MoE Inference with Adaptive Expert Offloading
AdaOffload is presented, a general and efficient multi-GPU hybrid expert execution framework that effectively handles heterogeneous hardware and workloads and co-designs GPU-CPU parallelism and GPU-CPU co-execution for decode and prefill with online workload observation that adapts to different hardware, workload, and...