Unmixing the Latent: Factor-Aware Sparse Adaptation for Vision-Language Models
TL;DR
SFASA is proposed, an approach that leverages multimodal contrastive learning to recover invariant semantic factors and dynamically routes only semantically salient tokens into lightweight low-rank adapters by reusing frozen VLM projection heads as zero-parameter modality mappers.
Abstract
Large pre-trained Vision-Language Models (VLMs) excel in zero-shot generalization but face challenges in efficient few-shot adaptation. Existing parameter-efficient methods apply uniform updates across all tokens, overlooking the inherent sparsity of cross-modal interactions and exacerbating overfitting. We propose SFASA: Semantic Factor-Aware Sparse Adaptation (SFASA), an approach that leverages multimodal contrastive learning to recover invariant semantic factors and dynamically routes only semantically salient tokens into lightweight low-rank adapters. By reusing frozen VLM projection heads as zero-parameter modality mappers, SFASA preserves pre-trained cross-modal priors while enabling targeted task-specific refinement. Extensive experiments on 11 datasets across four evaluation protocols show that SFASA consistently outperforms state-of-the-art methods in base-to-novel generalization, cross-dataset transfer, domain generalization, and few-shot learning.