The experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Abstract
Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation, and a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters.
Peng Liu, Hui-Bing Zeng, Yi-Qun Zhang et al.· IEEE Transactions on Emergin...· 0 citations
As pretrained language models grow, full parameter updates become prohibitively expensive, motivating parameter-efficient fine-tuning (PEFT). LoRA decomposes weight updates into low-rank factors, but its per-module cost $r(m{+}n)$ imposes a structural floor at rank one. FourierFT breaks this floor via frequency-domain...
Sayed Muhsin, Seok-Bum Ko· IEEE Signal Processing Lette...· 1 citation
Circuit Fine-Tuning is introduced, a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training to isolate the response of the backbone to the target distribution rather than the preferences of a particular classifier.
SPIRA is a PEFT framework that separates support discovery from parameter-efficient adaptation and remains competitive with representative PEFT baselines while using lower trainable-parameter budgets than several compared weight-side/LoRA-family baselines.
Zhong-Yi Wen, Zhikai Zhai, Guo-Min Sun et al.· IEEE Transactions on Pattern...· 0 citations
Although parameter-efficient fine-tuning significantly reduces the computing cost of deep models, default configurations are insufficient to perform as good as full fine-tuning for challenging large-cardinality intent detection problems with 77-151 intents. Thus, this work presents the S1 configuration that is proposed...
Harsh Anand, Sofia Singh, Rahul Agrawal et al.· 2026 International Conferenc...· 0 citations
This paper investigates SoRA algorithm's performance in large-scale models, and derives an improved learning rate configuration and proposes SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally and mitigates the suboptimality of the original SoRA method.
