Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
This work regularizes the energy of each rank-one LoRA component, encouraging redundant components to vanish while preserving important ones, and proposes a principled rank-allocation method based on classical sparsity-inducing technique in signal processing and statistics.
Ze-Bang Xie, Chuan-Yang Zheng, Yik-Chung Wu et al.
· 0 citations