Shape: Shaping The Low-Rank Subspace With Hierarchical Priors
Abstract
The escalating scale of Large Language Models (LLMs) has made Parameter-Efficient Fine-Tuning (PEFT) an essential paradigm for localized adaptation. However, standard methods like Low-Rank Adaptation (LoRA) typically rely on a static and uniform rank distribution across all layers. This structural agnosticism neglects the inherent heterogeneity of Transformer architectures, specifically the semantic processing role of middle layers, and often leads to rank collapse and parameter inefficiency. To address these limitations, we propose SHAPE, a novel adaptation framework that harmonizes architectural inductive biases with data-driven evolution. Our method integrates three synergistic components: Hierarchical Prior Injection, which employs a sinusoidal prior to strategically over-provision the rank budget for critical layers; Dynamic Rank Rectification, which acts as a corrective layer to refine the architectural hypothesis based on gradient sensitivity; and Subspace Orthogonality Regularization, which explicitly penalizes feature correlation to ensure geometric diversity within the low-rank subspace. Extensive experiments on Qwen2-7B-Instruct demonstrate that SHAPE utilizes approximately 20% fewer trainable parameters than standard LoRA while achieving superior performance on NLU tasks, validating that non-uniform, structured parameter allocation significantly enhances complex reasoning capabilities under strict budget constraints.