Adaptability assessment and optimal strategy selection for parameter-efficient fine-tuning of large language models
TL;DR
PEFT-Assess is presented, a systematic framework for evaluating the adaptability of four mainstream PEFT methods—LoRA, QLoRA, Adapter, and Prefix-Tuning—across five evaluation dimensions: accuracy, F1-score, training speed, memory efficiency, and generalization capability.
Abstract
Parameter-efficient fine-tuning (PEFT) methods have become essential for adapting large language models (LLMs) to downstream tasks without updating all parameters. However, the optimal PEFT strategy varies significantly across model architectures, task types, and resource constraints, making method selection a non-trivial problem. This paper presents PEFT-Assess, a systematic framework for evaluating the adaptability of four mainstream PEFT methods—LoRA, QLoRA, Adapter, and Prefix-Tuning—across five evaluation dimensions: accuracy, F1-score, training speed, memory efficiency, and generalization capability. We benchmark these methods on LLaMA-7B and Qwen-7B across six NLP tasks from the GLUE and SuperGLUE benchmarks. Furthermore, we propose a gradient boosting-based strategy selector that predicts the optimal PEFT method given task characteristics and resource budgets. Experiments show that LoRA achieves the best overall adaptability score of 0.847, while our strategy selector attains 91.2% prediction accuracy for recommending the optimal PEFT method.