Unmixing the Latent: Factor-Aware Sparse Adaptation for Vision-Language Models
SFASA is proposed, an approach that leverages multimodal contrastive learning to recover invariant semantic factors and dynamically routes only semantically salient tokens into lightweight low-rank adapters by reusing frozen VLM projection heads as zero-parameter modality mappers.