Nov 2026· Proceedings of the 34th ACM International Conference on Multimedia· pp. 5700-5708· 0 citations
Abstract
Large pre-trained Vision-Language Models (VLMs) excel in zero-shot generalization but face challenges in efficient few-shot adaptation. Existing parameter-efficient methods apply uniform updates across all tokens, overlooking the inherent sparsity of cross-modal interactions and exacerbating overfitting. We propose SFASA: Semantic Factor-Aware Sparse Adaptation (SFASA), an approach that leverages multimodal contrastive learning to recover invariant semantic factors and dynamically routes only semantically salient tokens into lightweight low-rank adapters. By reusing frozen VLM projection heads as zero-parameter modality mappers, SFASA preserves pre-trained cross-modal priors while enabling targeted task-specific refinement. Extensive experiments on 11 datasets across four evaluation protocols show that SFASA consistently outperforms state-of-the-art methods in base-to-novel generalization, cross-dataset transfer, domain generalization, and few-shot learning.
D2MoE is proposed, a Mixture of Task Adapters that explicitly factorizes these roles within a frozen DINOv3 backbone, establishing a principled and scalable route to prior-preserving, foundation-centric multi-task dense perception.
Yun-Zhi Zhuge, Xin-Zhuo Yu, Si-Tong Gong et al.· IEEE Transactions on Pattern...· 0 citations
ProCAP is proposed, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine...
H. Abbas, Fatemeh Daneshfar, M. Abdar· arXiv.org· 0 citations
Contrastive Language-Image Pre-training (CLIP) performs well on vision-language recognition tasks, but its generalization to novel classes may degrade when it is adapted to a specific dataset with only a few labeled base-class examples. Many recent CLIP-based methods address this problem by introducing learnable prompt...
Ying Dong, Xiao-Dong Gu· International Conference on...· 0 citations
Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redundancy or text-vision attention. However, they frequently suffer from semantic degradation due to their task-agnostic design or unreliable attention estimates. Based on ou...
Shuo Yang, Chang-Bai Li, Lin-Lin Yang et al.· 0 citations
Few-shot adaptation of vision-language models must balance two competing goals: learning fine-grained visual distinctions from very limited supervision and preserving the efficiency that makes frozen CLIP-like models attractive for deployment. Existing lightweight adaptation methods, including prompt tuning, LoRA, and...
Mohammed Rahman Sherif Khan Mohammad, Ardhendu Behera, Sandip Pradhan et al.· 0 citations
Generative vision-language models (VLMs) such as Qwen-VL and LLaVA achieve strong zero-shot performance on tasks overlapping with their pretraining distribution, yet fail on specialized domains where the required discriminative features were never learned, a regime we term distant out-of-distribution (OOD). Standard ad...
Hung-Jen Chen, Yuek F. Ho, Ting-Yao Huang et al.· European Conference on Compu...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.