Skip to content
Conference Open access

Shape: Shaping The Low-Rank Subspace With Hierarchical Priors

Sep 2026 · 2026 23rd International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · pp. 1-6 · 0 citations · 25 references

Abstract

The escalating scale of Large Language Models (LLMs) has made Parameter-Efficient Fine-Tuning (PEFT) an essential paradigm for localized adaptation. However, standard methods like Low-Rank Adaptation (LoRA) typically rely on a static and uniform rank distribution across all layers. This structural agnosticism neglects the inherent heterogeneity of Transformer architectures, specifically the semantic processing role of middle layers, and often leads to rank collapse and parameter inefficiency. To address these limitations, we propose SHAPE, a novel adaptation framework that harmonizes architectural inductive biases with data-driven evolution. Our method integrates three synergistic components: Hierarchical Prior Injection, which employs a sinusoidal prior to strategically over-provision the rank budget for critical layers; Dynamic Rank Rectification, which acts as a corrective layer to refine the architectural hypothesis based on gradient sensitivity; and Subspace Orthogonality Regularization, which explicitly penalizes feature correlation to ensure geometric diversity within the low-rank subspace. Extensive experiments on Qwen2-7B-Instruct demonstrate that SHAPE utilizes approximately 20% fewer trainable parameters than standard LoRA while achieving superior performance on NLU tasks, validating that non-uniform, structured parameter allocation significantly enhances complex reasoning capabilities under strict budget constraints.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.