ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning
Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singul...
Fan-Yao Li, Yu-Xi Liu, Xiao-Hao Liu et al.
· 0 citations