Parameter-efficient fine-tuning like low-rank adapters is widely used to adapt multimodal large language models to downstream tasks, yet we identify an unaddressed failure mode called Non-Textual Norm Collapse (NTNC). NTNC causes non-textual modality representations to shrink to near-zero after low-rank projection, lea...
Ze-Xiang Li, S. Zhou· Proceedings of the 34th ACM...· 0 citations
SFASA is proposed, an approach that leverages multimodal contrastive learning to recover invariant semantic factors and dynamically routes only semantically salient tokens into lightweight low-rank adapters by reusing frozen VLM projection heads as zero-parameter modality mappers.
Ze-Xiang Li, Wen-Xu Wang, Jia-Wen He et al.· Proceedings of the 34th ACM...· 0 citations
Large pre-trained Vision-Language Models (VLMs) excel in zero-shot generalization but face challenges in efficient few-shot adaptation. Existing parameter-efficient methods apply uniform updates across all tokens, overlooking the inherent sparsity of cross-modal interactions and exacerbating overfitting. We propose SFA...
Ze-Xiang Li, Wen-Xu Wang, Jia-Wen He et al.· Proceedings of the 34th ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.