Experimental results across three models across both domain-specific and general benchmarks demonstrate that NGFT significantly outperforms existing mainstream fine-tuning methods in both efficiency and performance, while effectively mitigating catastrophic forgetting.
Abstract
Existing Supervised Fine-Tuning paradigms, particularly Full Parameter Fine-Tuning are often plagued by parameter redundancy, inconsistent data quality, and catastrophic forgetting, which current methods typically address in isolation and lack a unified optimization signal to bridge data selection, parameter updates, and knowledge preservation. To address this, we propose Neuron-Guided Fine-Tuning (NGFT), a holistic framework that leverages neuron activation patterns as a universal proxy to unify the fine-tuning lifecycle. NGFT operates via three synergistic mechanisms: (1) Adaptive Task-Specific Neuron Selection, which identifies essential neurons in a single forward pass to concentrate updates and reduce redundancy; (2) Activation-Based Data Selection, which prioritizes information-dense samples that maximize contribution to key neurons; and (3) Neuron Activation Alignment, a novel loss function that anchors activations to pre-trained states, deepening representation learning and preserving general knowledge. Experimental results across three models across both domain-specific and general benchmarks demonstrate that NGFT significantly outperforms existing mainstream fine-tuning methods in both efficiency and performance, while effectively mitigating catastrophic forgetting.
GLA-LoRA establishes a unified learning strategy that synergistically integrates multi-granular contrastive learning with knowledge distillation and establishes that explicit global-local knowledge alignment is essential for achieving high-fidelity, parameter-efficient fine-tuning across diverse language tasks.
PRiSM is proposed, a training-free approach for layer-wise partial fine-tuning that leverages the cosine similarity between pre-trained aggregate token representations across layers to identify inter-layer relationships and achieves competitive performance compared to full fine-tuning.
A. Biswas, Md Fahim, M. Amin et al.· International Conference on...· 0 citations
DyPAM (Dynamic Positional Attention Modulation) is proposed, a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the...
Dayan Pan, Jing-Yuan Wang, Xie Yu· Proceedings of the 32nd ACM...· 0 citations
RaMod, a Representation-Aware Modularity framework, is proposed, a Representation-Aware Modularity framework to extend the ReFT paradigm to CTG through two novel components: Dual-Modular Representation & Parameter Fine-tuning, which manipulates only a strategically chosen subset of hidden representations with modular i...
Zheng Gong, Ying Sun, Chao Wang et al.· Proceedings of the Thirty-Fi...· 0 citations
This work argues that cross-modal alignment is implicitly captured in the information-compression trajectory, and proposes LLaVAFlow, an information-theoretic distillation framework that preserves alignment flow and enhances both downstream performance and generalization.
Mu-Yao Yuan, Mu-Yan Jiao, Jiang-Yong Ying et al.· 0 citations
Activation-Weighted Adaptive REtention (AWARe) is a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns, and operates without modifying model architectures, ensuring compatibility with existing inference engines.
Jun-Yong Liao, Jinfan Lv, Guoming Wang et al.· 1 citation