SVRA: Singular-Value Residual Adaptation for VLMs
Abstract
Contrastive Language-Image Pre-training (CLIP) performs well on vision-language recognition tasks, but its generalization to novel classes may degrade when it is adapted to a specific dataset with only a few labeled base-class examples. Many recent CLIP-based methods address this problem by introducing learnable prompts or additional adaptation modules, which bring extra design choices and may affect the balance between base-class fitting and novel-class generalization. In this work, we introduce Singular-Value Residual Adaptation (SVRA), a prompt-free and parameter-efficient adaptation method based on Singular Value Decomposition (SVD). SVRA applies SVD-initialized low-rank residual updates to the Multi-Head Attention (MHA) projection layers in selected Transformer layers, while keeping the principal pre-trained components, the remaining backbone parameters, and the Multi-Layer Perceptron (MLP) layers frozen. This design preserves most of the pre-trained representation while still giving the model limited but effective flexibility to adapt to a new task. We evaluate SVRA under the base-to-novel generalization setting on 11 datasets. SVRA achieves an average harmonic mean (HM) of 80.49%, improving over PromptSRC+DPC by 0.45 percentage points and the closely related CLIP-SVD baseline by 0.36 percentage points. The results show a favorable trade-off between adaptation performance and generalization, without prompt tuning or additional MLP-side modules.