Less is MoE: Trimming Experts in Domain-Specialist Language Models
Fisher-MoE is proposed, which operates within FFN to remove intermediate dimensions ranked by Fisher importance, and suggests intermediate dimension granularity is an effective unit for both compression and ranking where capability concentrates in MoE models.