Zhikai Zhai

SPIRA: Sparse Information-Geometric Rank Adaptation for Parameter-Efficient Fine-Tuning of Large Pretrained Models.

Downstream adaptation of large pretrained models (LPMs) via full-parameter fine-tuning is computationally prohibitive. Parameter-efficient fine-tuning (PEFT) methods, such as the widely used Low-Rank Adaptation (LoRA), reduce this cost but still parameterize dense updates over the selected weight matrices. This support...