Parameter-Efficient Fine-Tuning of BERT: A Study of Fixed-Rank and Layer-Wise LoRA on SST-2

Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as a crucial method for adapting pretrained Transformer models to downstream tasks without the need to fine-tune the full set of model parameters. Low-Rank Adaptation (LoRA) is a technique that reduces the number of trainable parameters by representing weight updates with low-rank matrices. This paper explores the impact of LoRA rank selection and layer-wise rank allocation on the trade-off between classification accuracy and the number of parameters to be fine-tuned through fine-tuning BERT-base-uncased on the sentiment classification task, SST-2. The three fixed-rank LoRA settings with ranks of 4, 8, and 16 are first investigated, as well as a manually-designed layer-wise LoRA setting that assigns different ranks to different subsets of BERT layers. In the fixed-rank experiments, increasing the rank is beneficial to the validation accuracy, but the magnitude of the improvement seems to diminish with the number of trainable parameters. The layer-wise setup is able to attain 92.20% validation accuracy with 345,602 trainable parameters. It decreases the number of trainable parameters by approximately 41.6% and improves the validation accuracy by 0.57 percentage point, compared with LoRA at r = 16. It also can achieves performance close to full fine-tuning with 109,483,778 trainable parameters and 92.32% accuracy. The results indicate that with a non-uniform rank allocation it is possible to achieve an effective compromise between the efficiency of the parameters and the goodness of the downstream classification.