Aug 2026· Neural Networks· Vol 205 Pt B, pp.
109545
· 0 citations· 40 references
Medicine
TL;DR
GLA-LoRA establishes a unified learning strategy that synergistically integrates multi-granular contrastive learning with knowledge distillation and establishes that explicit global-local knowledge alignment is essential for achieving high-fidelity, parameter-efficient fine-tuning across diverse language tasks.
Abstract
The rapid advancement of large language models (LLMs) has revolutionized natural language processing, yet their enormous parameter size presents significant challenges for fine-tuning on downstream tasks. While parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA) have substantially reduced computational requirements, they often introduce knowledge misalignment and representation degradation during adaptation. To bridge this gap, we propose GLA-LoRA, a novel PEFT framework centered on Global-Local Knowledge Alignment. Our approach establishes a unified learning strategy that synergistically integrates multi-granular contrastive learning with knowledge distillation. GLA-LoRA explicitly preserves dataset-wide semantic structures through global contrastive learning and maintains sample-level feature consistency via local invariance learning, while a teacher model guides the adaptation to retain the original model's knowledge. Extensive evaluations across eight benchmarks, spanning six GLUE tasks and two intent detection datasets, demonstrate that GLA-LoRA consistently outperforms strong baselines including standard LoRA and QLoRA. Under identical parameter budgets, GLA-LoRA achieves consistent and significant improvements over standard LoRA across three base models and eight benchmark datasets. On sentence-level classification and regression tasks, our method yields average accuracy gains of 1.4%, 1.1%, and 1.2% on LLaMA2-7B, RoBERTabase, and BERTLarge, respectively. For intent detection (CLINC and HWU), it achieves even more pronounced improvements, with up to 2.3% accuracy gain on BERTLarge. These results establish that explicit global-local knowledge alignment is essential for achieving high-fidelity, parameter-efficient fine-tuning across diverse language tasks.
Modern natural language systems rely on large language models, whose sheer size makes full fine-tuning costly in computation, graphics processing unit (GPU) memory, and storage. Low-rank adaptation (LoRA) sidesteps most of that cost. It keeps the pre-trained weights frozen and captures each task-specific change as the product of two smaller matrices, so adapting a model reduces to a low-rank decomposition. This review covers LoRA and its main variants and pays particular attention to the linear algebra behind them. It first explains why the low intrinsic dimension of fine-tuning makes low-rank updates effective, then compares the major variants: quantized LoRA (QLoRA), quantization-aware LoRA (QA-LoRA), adaptive low-rank adaptation (AdaLoRA), sparse low-rank adaptation (SoRA), and weight-decomposed low-rank adaptation (DoRA). Across published studies, these methods come close to full fine-tuning accuracy while updating well under one percent of a model's parameters in some settings. For reference, LoRA cuts the trainable parameter count of Generative Pre-trained Transformer 3 (GPT-3) by four orders of magnitude, and QLoRA brings a 65-billion-parameter model within the memory of one 48 GB card. Open problems remain in choosing the rank, comparing results across studies, limiting quantization loss, and combining multiple adapters without interference. Ultimately, an established piece of linear algebra, approximating high-dimensional objects in low-dimensional subspaces, is what keeps the adaptation of very large models affordable.
Unknown authors· Theoretical and Natural Scie...· 0 citations
Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. However, the enormous size of LLMs poses significant challenges in terms of computational complexity and resource requirements. Low-Rank Adaptation (LoRA) has emerged as a promising solution, yet a gap remains between the practical performance of low-rank adaptations and their theoretical optimum. While recent works have explored iteratively merging LoRA adapters, they lack a principled theoretical framework to guide adapter design. In this work, we propose eXtreme Gradient Boosting LoRA (XGBLoRA), a novel framework grounded in gradient boosting theory. Our key insight is that the adapter must adhere to the weak learner principle-each individual adapter should have limited expressiveness---to ensure that the iterative ensemble can effectively raise the model's performance ceiling without overfitting. We provide theoretical analysis establishing convergence guarantees and expressiveness bounds, which formally justify why weaker (lower-rank) adapters, when properly combined, can match or exceed the performance of stronger (higher-rank) ones. Extensive experiments on natural language processing tasks demonstrate that XGBLoRA with rank-1 updates consistently outperforms standard LoRA with significantly fewer trainable parameters.
Yifei Zhang, Hao Zhu, Haoran Shi et al.· Proceedings of the 32nd ACM...· 0 citations
This work argues that cross-modal alignment is implicitly captured in the information-compression trajectory, and proposes LLaVAFlow, an information-theoretic distillation framework that preserves alignment flow and enhances both downstream performance and generalization.
Muyao Yuan, Muyan Jiao, Jiangyong Ying et al.· 0 citations
This work proposes a distillation approach based on ranking supervision that consistently outperforms supervised fine-tuning as well as FKL and RKL baselines in Python code generation, multilingual generation, and data-science scenarios and offers guidance for future research in model compression.
Zhe Ding, Hui Ji, Su Pan et al.· Neural Networks· 0 citations
LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure, improves overall performance over vanilla SFT and recent SFT-enhancement baselines, suggesting that local preservation helps mitigate capability degradation without collapsing sampling-accessible diversity.
Yueyang Wang, Baolong Bi, Shuo Lu et al.· 0 citations
TopoTuner is competitive with full fine-tuning while training only 1-2% of the model parameters, and outperforms LoRA in 7 out of 9 model-dataset settings, which can change up to 39.57% of the projection parameters.
Abdulkadir Erol, Yash Mahajan, Vepaul Hariprashad et al.· 0 citations