ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
The proposed ACE (Adapter Consolidation across Experts), which groups redundant experts and replaces their expert-specific adapters with group-shared higher-rank LoRA modules under the same PEFT budget, achieves the highest observed mean accuracy among the parameter-matched PEFT methods on the three backbones with comp...
Ahin Lee, Sehyun Yun, Joonha Park et al.
· 0 citations