Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Mixture-of-Experts (MoE) models activate few experts per token, yet their full expert sets can exceed GPU memory and require repeated weight transfers during decoding. We formulate expert-cache management as a model-side algorithmic problem and propose cache-aware post-training that jointly adapts the MoE backbone and...
Zhenhe Wu, Ya-Ping Jin, Qing-Hua Xing et al.
· 2 citations