Layer-wise Rank Allocation for Parameter-Efficient Fine-Tuning in Vision Transformers

Abstract

Vision Transformer (ViT) obtains state-of-the-art performance but is expensive to fine-tune because of the large number of parameters. Parameter-efficient tuning methods, such as LoRA, have been proposed. However, existing methods tend to ignore the hierarchical structure of representation learning and assume the same distributed-rank assignment for all layers. This paper offers a rank assignment technique for LoRA finetuning of Vision Transformer with layer-wise differential ranking rather than uniform, which gives better channel utilization efficiency. The test used the CIFAR-100 dataset with the ViT Tiny model. The testing results show that the method can achieve 0.8086 average accuracy while decreasing the tunable parameter ratio to 2.43% from the original 4.09%, which is an advantage over LoRA with its uniform rank. Additionally, the method is seed-agnostic, and thus, the approach is reasonably stable. These findings suggest that placing different hierarchies increases parameter efficiency; hence, it is a viable method that can be used to efficiently fine-tune Vision Transformers.