MoRE: Mixture of Reused Experts
Experiments show that MoRE consistently achieves lower perplexity and stronger downstream performance than standard MoEs and state-of-the-art weight-sharing architectures at matched compute and parameter budgets, with only minimal modifications to existing MoE implementations.
Eric S. Qiu, U. U. Acikalin, Justin Lovelace et al.
· 1 citation