In CoDA, a new adaptation framework that explicitly disentangles and coordinates cross-modal semantic alignment and intra-modal structural consistency is proposed, and it is shown that CoDA outperforms state-of-the-art parameter-efficient methods, particularly under few-shot learning and distribution-shift scenarios.
Vision-language models (VLMs), such as contrastive language-image pre-training (CLIP), exhibit powerful zero-shot generalization capabilities. Parameter-efficient fine-tuning (PEFT) techniques, notably prompt learning, have been extensively explored to adapt these models to downstream tasks. However, their efficacy remains constrained when transferred to specialized domains like remote sensing. We argue that the bottleneck stems not merely from the limited parameters of prompts, but essentially from the disruption of the input’s original image–text features and the lack of deep cross-modal alignment. In particular, existing methods typically rely on global attention or coarse-grained feature mapping. This inadvertently corrupts the original input representations, thereby impairing the model’s inherent generalization. Furthermore, their isolated unimodal gradient updates fail to bridge the semantic gap inherent in complex remote sensing scenes. To address these challenges, we propose tokenwise prompt-free learning (Tiper), shifting the optimization paradigm from introducing external prompts to precisely recalibrating the critical tokens that govern classification outputs. In particular, Tiper employs a hierarchical learner to supersede global prompts. Crucially, this learner intervenes exclusively on the specific core tokens (i.e., the CLS token in the visual branch and the EOT token in the textual branch), leaving other original input representations unperturbed. This fine-grained strategy effectively balances domain adaptation with the preservation of inherent generalization. Finally, we design the learner as a cross-modal coupled bridge with shared weights, enabling it to synchronously receive gradient feedback from both modalities and fostering profound multimodal collaboration. Extensive experiments validate our method on eight public remote sensing datasets covering diverse scenes and resolutions. In the base-to-new generalization task, Tiper outperforms the strong baseline MaPLe with a significant 3.7% improvement in the harmonic mean (HM). Notably, without relying on any external large-scale domain models, Tiper surpasses the latest domain-specific prompt learning methods (e.g., domain-controlled prompt learning (DCPL), domain prompt learning with quaternion networks (DPLQ)), demonstrating its superior adaptability for remote sensing image scene classification.
Tengfei Gong, Jun-Lin Wu, Yaxioong Chen et al.· IEEE Transactions on Geoscie...· 0 citations
Few-shot learning with vision-language models suffers from a fundamental structural limitation: support and query samples are processed through independent and asymmetric encoding pipelines. This causes query-side semantic blindness, where the model lacks rich cross-modal interactions during query encoding. Consequently, it weakens vision-language alignment and creates a training-inference distribution gap, degrading generalization to novel categories. Existing prompt-based methods inherit this asymmetry and thus cannot leverage text-conditioned semantic context on the query side at inference time. To address this limitation, we propose an Adaptive Cross-Modal Alignment via Symmetric Prompt Tuning for Few-Shot Vision–Language Learning (ACAS-PT) a unified framework that resolves this issue via symmetric prompt tuning. ACAS-PT applies identical prompt-guided, text-conditioned feature transformations to both support and query samples in a shared multimodal space, eliminating the distribution gap by design. Specifically, we propose two modules. First, a Semantic-Aware Class-Embedding Learner transforms prompt-conditioned CLIP class embeddings into class-specific semantic vectors used to modulate both support and query visual features via FiLM-based affine transformation, ensuring that query samples receive the same class-specific semantic grounding as support prototypes at inference. Second, an Adaptive Similarity Guided Module (ASGM) replaces fragile equal-weight prototype averaging with learnable instance-weighted centroid aggregation and a per-class-pair cross-modal alignment matrix that gates classification scores by within-class semantic-visual alignment confidence, yielding robust prototype estimates even under extreme label scarcity. Extensive experiments on four benchmark datasets show ACAS-PT outperforms 16 state-of-the-art methods, with symmetric processing alone yielding up to a +2.5% improvement in 5-shot accuracy. These results highlight query-side semantic blindness as a critical bottleneck in vision-language few-shot learning.
Feidu Akmel, Xun Gong, Abdalla Hadabi et al.· Journal of King Saud Univers...· 0 citations
This work proposes a principled VLM TTA method called \algname, and theoretically reveals that the InfoNCE loss can be neatly reformulated as a Wasserstein OT formulation, thereby unifying the objectives of the inference and adaptation of VLMs to achieve their mutual benefits.
Qi Yu, Zhichen Zeng, Katherine Tieu et al.· 0 citations
The results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.
Xuanru Zhou, Yiwen Shao, Jiahong Li et al.· 1 citation
This work argues that cross-modal alignment is implicitly captured in the information-compression trajectory, and proposes LLaVAFlow, an information-theoretic distillation framework that preserves alignment flow and enhances both downstream performance and generalization.
Muyao Yuan, Muyan Jiao, Jiangyong Ying et al.· 0 citations
The Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP) is proposed, enhancing both the robustness and granularity of vision-language alignment and underscore RoFLIP’s compositional reasoning and generalization abilities.
Yiwei Sun, Chuanbin Liu, Shancheng Fang et al.· International Journal of Com...· 0 citations
