From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
NSFT (Neural Sub-expert Fine-Tuning), a fine-grained PEFT framework that refines MoE adaptation from experts to sub-experts, is proposed and suggests that sub-expert-level adaptation is a more precise and efficient PEFT paradigm for MoE LLMs.
Zhen-Tao Tan, Chang Liu, Yao Liu et al.
· 1 citation