One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module he...
Hui-Yi Wang, Daijiao Liu, Lina Yao et al.
· 0 citations