This study provides a technical pathway and empirical evidence for deploying large models in the construction domain under resource-constrained conditions through the synergy of fine-tuning and RAG.
Abstract
Addressing the resource constraints of a single NVIDIA RTX 5000 (16 GB) GPU, this applied study takes DeepSeek-LLM-7B-Base as the base model and systematically compares four parameter-efficient fine-tuning methods: LoRA, QLoRA, DoRA, and LoRA+. It also validates a Retrieval-Augmented Generation (RAG) architecture tailored for zero-tolerance engineering specifications. Experiments are conducted on a private construction process dataset. Theoretical analysis shows that the low-rank assumption of LoRA originates from the intrinsic dimensionality property of pre-trained models; LoRA+ adopts an asymmetric learning rate strategy (with the optimal ratio ηAηB = 0.05 determined via grid search), effectively solving the suboptimal training dynamics problem of standard LoRA caused by a uniform learning rate; DoRA decomposes weight updates into magnitude and direction components on a spherical manifold and a positive real manifold; RAG guarantees hallucination suppression through the conditional entropy inequality H(Y|Q,D,θ) ≤ H(Y|Q,θ). Experimental results demonstrate that LoRA+ outperforms other baseline methods in BLEU-4 (0.5609), ROUGE-L (0.5387), and PPL (2.1433), with a training time of 1.8 h and memory usage of 13.1 GB. After introducing RAG on top of LoRA+, BLEU-4 further improves to 0.5814, ROUGE-L to 0.5557, and the hallucination rate(HR) drops from 1.71% to 0.08%, achieving an Exact Match (EM) score of 0.2778 and high traceability (Recall@3 = 0.9961). This study provides a technical pathway and empirical evidence for deploying large models in the construction domain under resource-constrained conditions through the synergy of fine-tuning and RAG.
Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $\Delta W$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Harshavardhan Adepu, Li Zhang, Sanjiv Kumar et al.· 0 citations
PoLoRA is introduced, a Preconditioned Orthogonalized LoRA optimizer built from three ingredients: a product-aware spectral update direction, curvature preconditioning derived from controlling the per-sample loss change, and a magnitude rule that controls the sizes of both the factor and merged updates.
Nikhil Ghosh, Tetiana Parshakova, Robert M. Gower· 1 citation
Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.
Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies. We propose \textbf{SAD-LoRA} (\textbf{S}pectral \textbf{A}lignment \textbf{D}istillation), which selects this subspace from the data-weighted student-space reference update $\DWT\Sigx^{1/2}$ and maintains it during training via a differentiable principal-angle loss on $\colspan(B)$. We show that the data-weighted distillation error decomposes exactly into subspace misalignment, within-subspace coefficient mismatch, and irreducible rank residual; standard KD can affect the first term only indirectly through output gradients. On controlled synthetic problems with a flat teacher spectrum, SAD-LoRA reduces the subspace-misalignment term from $51\%$ to nearly zero and lifts final subspace alignment from $0.49$ to $1.00$. On RoBERTa-large to RoBERTa-base distillation across six GLUE tasks, SAD-LoRA improves rank efficiency: at $r{=}4$, it matches or beats the strongest included spectral baseline on five of six tasks, and at $r{=}8$ it gives the best result on SST-2 and CoLA. Ablations identify subspace alignment as the load-bearing component, while coefficient matching is auxiliary.
Omer Tariq, Syed Muhammad Raza, J. Son· 0 citations
A unified pipeline deployed at Walmart that addresses both signal quality and model evolution is presented, and a Warm-Start Distillation technique that transfers domain-specific expertise from the legacy model to the new backbone is introduced.
Zhen Yang, Juexin Lin, Hongwei Shang et al.· Annual International ACM SIG...· 1 citation
GLA-LoRA establishes a unified learning strategy that synergistically integrates multi-granular contrastive learning with knowledge distillation and establishes that explicit global-local knowledge alignment is essential for achieving high-fidelity, parameter-efficient fine-tuning across diverse language tasks.