Skip to content
Book Open access

Unmixing the Latent: Factor-Aware Sparse Adaptation for Vision-Language Models

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · pp. 5700-5708 · 0 citations

Abstract

Large pre-trained Vision-Language Models (VLMs) excel in zero-shot generalization but face challenges in efficient few-shot adaptation. Existing parameter-efficient methods apply uniform updates across all tokens, overlooking the inherent sparsity of cross-modal interactions and exacerbating overfitting. We propose SFASA: Semantic Factor-Aware Sparse Adaptation (SFASA), an approach that leverages multimodal contrastive learning to recover invariant semantic factors and dynamically routes only semantically salient tokens into lightweight low-rank adapters. By reusing frozen VLM projection heads as zero-parameter modality mappers, SFASA preserves pre-trained cross-modal priors while enabling targeted task-specific refinement. Extensive experiments on 11 datasets across four evaluation protocols show that SFASA consistently outperforms state-of-the-art methods in base-to-novel generalization, cross-dataset transfer, domain generalization, and few-shot learning.

Read PDF

Similar papers

Open access Nov 2026

Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters

D2MoE is proposed, a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.

Yun-Zhi Zhuge, Xin-Zhuo Yu, Si-Tong Gong et al. · 0 citations

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

ProCAP is proposed, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine...

H. Abbas, Fatemeh Daneshfar, M. Abdar · 0 citations
Conference Open access Oct 2026

SVRA: Singular-Value Residual Adaptation for VLMs

Contrastive Language-Image Pre-training (CLIP) performs well on vision-language recognition tasks, but its generalization to novel classes may degrade when it is adapted to a specific dataset with only a few labeled base-class examples. Many recent CLIP-based methods address this problem by introducing learnable prompt...

Ying Dong, Xiao-Dong Gu · 0 citations
#machine learning Preprint Oct 2026

DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models

Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redundancy or text-vision attention. However, they frequently suffer from semantic degradation due to their task-agnostic design or unreliable attention estimates. Based on ou...

Shuo Yang, Chang-Bai Li, Lin-Lin Yang et al. · 0 citations
#artificial intelligence Preprint Nov 2026

SD-LoRA: Training-Time Structural Distillation into LoRA for Few-Shot Vision-Language Adaptation

Few-shot adaptation of vision-language models must balance two competing goals: learning fine-grained visual distinctions from very limited supervision and preserving the efficiency that makes frozen CLIP-like models attractive for deployment. Existing lightweight adaptation methods, including prompt tuning, LoRA, and...

Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan et al. · 0 citations
Conference Open access Sep 2026

Inductive Visual Logic for Few-Shot Out-Of-Distribution Adaptation in VLMs

Generative vision-language models (VLMs) such as Qwen-VL and LLaVA achieve strong zero-shot performance on tasks overlapping with their pretraining distribution, yet fail on specialized domains where the required discriminative features were never learned, a regime we term distant out-of-distribution (OOD). Standard ad...

Hung-Jen Chen, Yuek F. Ho, Ting-Yao Huang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.