Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models
TL;DR
DyPAM (Dynamic Positional Attention Modulation) is proposed, a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the pretrained backbone.
Abstract
Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing PEFT methods rely on uniform and static adaptations, without accounting for the structured heterogeneity of attention across dimensions, heads, layers, and input tokens. In practice, attention representations exhibit non-uniform behavior, and positional encoding mechanisms such as rotary positional embeddings (RoPE) induce dimension-dependent positional structure, making uniform adaptation suboptimal. In this work, we propose DyPAM (Dynamic Positional Attention Modulation), a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations. DyPAM combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the pretrained backbone. Extensive experiments on mathematical and commonsense reasoning benchmarks across multiple backbone models demonstrate that DyPAM consistently outperforms existing strong PEFT baselines. The code is available for reproducibility at https://github.com/Beihang-BIGSCity/DyPAM.