PEFT unlocked: methodologies, formulas, and applications in efficient LLM adaptation with a specific focus on LoRA

A Unified Taxonomy of Deep Learning Optimizers for Scalable and Efficient AI Systems

A decision-oriented framework that translates theoretical insights into practical guidelines is introduced, establishing a structured foundation for broader communities to train and deploy human-centered AI systems sustainably and efficiently.

Design of Resource-Efficient AI Models through Parameter Reduction and Accuracy-Aware Compression

The proposed hybrid pipeline includes structured pruning, INT8 quantization and task-specific knowledge distillation, which is benchmarked against standalone methods and reinforces the idea of upper bound projection based approach for accuracy-oriented, multi-level compression.

DeepSeek-V3: Architecture and Optimizations-A Practical Review

The design of transformer-based Large Language Models (LLMs) is being radically changed through new architectures that are able to overcome scalability limitations of previous designs, including Mixture-of-Experts (MoE), Multi-Head Latent Attention (MLA), and Multi-Token Prediction (MTP). As an open-weighted model released at the end of 2024, which has both state of the art architectural transparency and production scale efficiency, DeepSeeek-V3 represents the ultimate testing ground for investigating these modern technologies. This paper provides a comprehensive analysis of the architectural structure of DeepSeek-V3 based upon information from the DeepSeek-V3 Technical Report, industry benchmarking data and independent latency testing, to demonstrate how various techniques can be used to optimize training while still providing competitive performance in code generation and mathematical reasoning. In addition, latency testing conducted on a Distilled version of DeepSeek-V3, with approximately 14 billion parameters, running on a T4 GPU, reveals that although significant improvements have been made in optimizing latency there remains substantial barriers to deploying these models. Through this context, this research will serve as a reference document for practitioners and researchers who wish to understand current trends and challenges in increasing accessibility to high performance AI models.

Research on Efficient Fine-Tuning of Large Model Parameters Using a Hybrid LoRA and IA3 Adaptation Approach

Addressing the high computational cost, large GPU memory consumption, and limited cross-task generalization of full-parameter fine-tuning for Large Language Models (LLMs), especially when adapting them to specialized engineering domains such as electromagnetic wave analysis, antenna design, and propagation scenario modelling, this paper proposes a parameter-efficient fine-tuning method named LoRA-IA3. The method combines Low-Rank Adaptation (LoRA) with Infused Adapter by Inhibiting and Amplifying Inner Activations (IA3), and achieves task adaptation while preserving the general knowledge of the pre-trained model through a dual mechanism of low-rank matrix decomposition and activation inhibition-amplification. First, the LoRA module is introduced into the attention layer of the Transformer architecture to perform low-rank decomposition on key weight matrices, thereby reducing the number of trainable parameters. Second, the IA3 adapter is embedded in the Feed-Forward Network (FFN) layer to dynamically adjust activation distributions through per-channel scaling factors, enhancing task-specific feature extraction. Finally, to maintain stable training and fast convergence when adapting models to complex electromagnetic engineering tasks, including antenna parameter interpretation, propagation data analysis, and technical text understanding, a hybrid adaptation training strategy is designed using hierarchical learning rates and gradient clipping mechanisms across different modules. The proposed approach provides an efficient fine-tuning framework for applying LLMs to domain-specific engineering tasks with reduced computational overhead.

Fisher-Based Layer-Wise Adaptive Sparsification for Efficient Pruning of Large Language Models

Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation

With the rapid development of large-scale pre-trained language models based on Transformer architectures, their high computational and memory costs have become a major obstacle to deployment, especially in resource-constrained environments. Traditional pruning methods typically depend on full gradient-based importance estimation, and they necessitate prior finetuning of the model to achieve satisfactory performance. This process often results in intolerable resource consumption. This paper proposes REP-LIE, a new approach to enable resource-efficient pruning during the process of finetuning. REP-LIE leverages the gradients of LoRA low-rank matrices to estimate the importance of weights without requiring full gradient computation. To address the inherent randomness in importance estimation, a stability score is introduced, serving as the basis for iterative pruning of unimportant model parameters. The pruned model is further finetuned through lightweight updates, eliminating the need for full-parameter optimization in the process of finetuning. Extensive experiments on both medium-scale encoder models and large-scale generative models (LLaMA-7B and Mistral-7B) demonstrate that REP-LIE still achieves competitive performance compared to existing approaches.