Skip to content

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Sep 2026 · arXiv.org · Vol abs/2609.37027 · 0 citations
Computer Science

TL;DR

Experiments show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT, and incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations.

Abstract

Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-tuning (FFT). Many LoRA variants improve the initialization or optimization of low-rank factors. At each training step, however, their first-order weight-space directions are constrained by the current parameterization. We characterize the corresponding LoRA-accessible gradient space and show that it coincides with the tangent space induced by the current LoRA parameterization. This characterization yields an orthogonal decomposition of the full weight gradient at the current model parameters. We term the component orthogonal to this space the normal gradient. Based on this decomposition, we propose GDLoRA (Gradient-Decomposed Low-Rank Adaptation). GDLoRA reconstructs the full weight gradient from forward activations and backward signals, extracts its normal component, and directly updates the base weights with this component, while retaining standard AdamW optimization for the LoRA factors. GDLoRA incorporates complementary normal gradients without increasing standard LoRA's optimizer-state memory budget under matched adapter and optimizer configurations. Experiments on natural language understanding, mathematical reasoning, commonsense reasoning, and image classification show that GDLoRA consistently improves over LoRA and narrows the performance gap to FFT. The code is available at https://anonymous.4open.science/r/GDLoRA.

View source

Similar papers

#machine learning Preprint Oct 2026

DT-LoRA: Reaching Beyond the Tangent Space of Low-Rank Adaptation

Low-rank adaptation (LoRA) often underperforms full fine-tuning. We investigate this performance gap through a tangent-space lens: factorized updates are confined to the tangent space of the current low-rank factors, leaving gradient components outside this space unexploited. An analysis of ten representative LoRA vari...

Yang-Ming Zhang, Ling Yi, Da-Tong Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

TaRA: Training-Aware Low-Rank Adaptation Initialization

Training-aware Low-Rank Adaptation Initialization (TaRA) is proposed, a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix.

Taehyeon Kim, Eunhyeok Park · 0 citations
#artificial intelligence Preprint Aug 2026

LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment

This paper introduces a lightweight probe for multi-step gradients of pretrained weights that incurs no additional GPU memory cost and only marginal time overhead, and employs a spectrum-aware, importance-based rank allocation and optimal initialization derived from multi-step gradients.

Hao-Nan He, Xin Fan · 0 citations
Conference Open access Sep 2026

SeMi-LoRA: Enhancing Low-Rank Adaptation via Separation and Mixing

This work proposes SeMi-LoRA (Separation and Mixing LoRA), a novel framework that enables complete high-rank updates while preserving mergeability and consistently outperforms strong PEFT baselines with favorable parameter efficiency.

Zhen-Fei Yang, Beiming Yu, Peiqin Lin et al. · 0 citations
#machine learning Preprint Sep 2026

Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representat...

Zi-Han Zhu, Zhe-Hang Du, Xu-Yang Chen et al. · 0 citations
Open access Sep 2026

TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-Tuning LLMs.

Zeroth-order optimization (ZO) has demonstrated remarkable promise in efficient fine-tuning tasks for Large Language Models (LLMs). In particular, recent advances incorporate the low-rankness of gradients, introducing low-rank ZO estimators to further reduce GPU memory consumption. However, most existing works focus so...

Y. Sun, Qi-Xin Zhang, Liang Ding et al. · 12 citations · ⚡1

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.