PE-CPT adapts a pre-trained vision-language model by learning a small set of prompt vectors without updating its original parameters, and incorporates LoRA, a parameter-efficient adaptation method applied to attention and feed-forward layers to improve adaptation.
Abstract
Ancient murals are invaluable cultural heritage, and their automatic classification is crucial for digital preservation, stylistic analysis, and heritage management. Existing approaches require large, high-quality annotated datasets, but such expert annotations are costly and often unavailable. To address this, we formulate a few-shot ancient mural classification problem and propose a Parameter-Efficient Cross-modal Prompt Tuning (PE-CPT) framework. Specifically, PE-CPT adapts a pre-trained vision-language model by learning a small set of prompt vectors without updating its original parameters. It performs cross-modal prompt tuning between image and text encoders, generating image prompts from text prompts to maintain semantic consistency. To further improve adaptation, we incorporate LoRA, a parameter-efficient adaptation method applied to attention and feed-forward layers. Experiments in a 16-shot setting demonstrate that PE-CPT outperforms state-of-the-art baselines, achieving improvements of 6.79%, 6.59%, and 7.07% in accuracy, precision, and F1 score. These results demonstrate its effectiveness in cultural heritage analysis under data scarcity.
Zero-shot industrial anomaly detection (ZIAD) aims to develop a unified model capable of directly identifying unseen anomaly categories in images without requiring reference samples. Recently, large-scale Vision-Language Models (VLMs) such as CLIP have shown great potential for solving this task. However, existing methods typically rely on manual text prompts to guide VLMs in anomaly detection, which often fail to capture fine-grained semantic cues, leading to limited accuracy. To address the challenge, this paper proposes a novel Cross-Modal Guidance Learning (CMGL) framework for ZIAD. Instead of handcrafted textual prompts, CMGL introduces learnable prompting mechanism to fully exploit the collaborative guidance between visual and textual modalities for efficient unseen anomaly detection. Leveraging the frozen image encoder of pre-trained CLIP, the CMGL extracts multi-scale patch tokens and global tokens of the input image as visual representations. Then, informed by the cross-modal information, adaptive prompt vectors are constructed to obtain textual representations. In the process, a Learnable Context Block (LCBlock) and a Multi-Layer Perceptron (MLP) are introduced to extract holistic semantics and fine-grained details, and an Adaptive State Vector Module (ASVM) is designed to learn generalized normal and abnormal state vectors from extensive text descriptions. By aggregating the outputs of these components, textual representations of the image are acquired through the frozen text encoder. Finally, a Local-Global Token Integrator (LGTI) and an Uncertainty-Aware Anomaly Fusion Module (UAFM) are proposed to achieve anomaly recognition and localization through visual–textual alignment. Extensive experiments on multiple industrial datasets demonstrate the superiority of our method. Note to Practitioners—This paper presents a Cross-Modal Guidance Learning (CMGL) framework to address anomaly detection under the zero-shot setting. Unlike previous approaches that rely on manually crafted text prompts, the proposed CMGL derives task-relevant prompt cues from cross-modal data by the designed learnable prompting mechanism, guiding the model to automatically recognize and localize unseen anomaly categories without requiring any reference samples. Extensive experiments demonstrate the effectiveness and strong generalization capability of the proposed approach. Benefiting from these properties, our method provides a novel and effective ZIAD solution for identifying potential anomalies in real-world industrial scenarios where data distributions are uncertain or anomaly-related information cannot be clearly specified. Our project page is publicly available at https://aicoder12.github.io/CMGL/
Tiyu Fang, Lin Zhang, Ran Song et al.· IEEE Transactions on Automat...· 0 citations
Few-shot learning with vision-language models suffers from a fundamental structural limitation: support and query samples are processed through independent and asymmetric encoding pipelines. This causes query-side semantic blindness, where the model lacks rich cross-modal interactions during query encoding. Consequently, it weakens vision-language alignment and creates a training-inference distribution gap, degrading generalization to novel categories. Existing prompt-based methods inherit this asymmetry and thus cannot leverage text-conditioned semantic context on the query side at inference time. To address this limitation, we propose an Adaptive Cross-Modal Alignment via Symmetric Prompt Tuning for Few-Shot Vision–Language Learning (ACAS-PT) a unified framework that resolves this issue via symmetric prompt tuning. ACAS-PT applies identical prompt-guided, text-conditioned feature transformations to both support and query samples in a shared multimodal space, eliminating the distribution gap by design. Specifically, we propose two modules. First, a Semantic-Aware Class-Embedding Learner transforms prompt-conditioned CLIP class embeddings into class-specific semantic vectors used to modulate both support and query visual features via FiLM-based affine transformation, ensuring that query samples receive the same class-specific semantic grounding as support prototypes at inference. Second, an Adaptive Similarity Guided Module (ASGM) replaces fragile equal-weight prototype averaging with learnable instance-weighted centroid aggregation and a per-class-pair cross-modal alignment matrix that gates classification scores by within-class semantic-visual alignment confidence, yielding robust prototype estimates even under extreme label scarcity. Extensive experiments on four benchmark datasets show ACAS-PT outperforms 16 state-of-the-art methods, with symmetric processing alone yielding up to a +2.5% improvement in 5-shot accuracy. These results highlight query-side semantic blindness as a critical bottleneck in vision-language few-shot learning.
Feidu Akmel, Xun Gong, Abdalla Hadabi et al.· Journal of King Saud Univers...· 0 citations
Pre-trained vision-language models (VLMs) like CLIP have achieved remarkable success in zero-shot visual recognition. While recent advancements leverage Large Language Models (LLMs) to generate fine-grained category descriptions in order to enhance CLIP-based models, they often suffer from significant spatial granularity mismatch (fine-grained category descriptions vs. the global image) and rely heavily on labor-intensive handcrafted prompt templates. To address these challenges, we propose the Adaptive Prompt-guided Local Cross-modal Alignment (AP-LCA) approach. Our approach introduces a novel local cross-modal alignment strategy that utilizes image cropping and similarity-based semantic contribution assessment to precisely map fine-grained descriptions to relevant local image regions. Additionally, we incorporate an in-context learning (ICL) mechanism to automate the generation of task-adaptive prompts, seamlessly evolving from manual templates to context-aware descriptions that capture diverse visual concepts. Extensive experiments across eight benchmark datasets demonstrate the clear superiority of our proposed method.
Siying Wu, Song Wu· International Conference on...· 0 citations
This work introduces a dual-branch ensemble framework fusing Semantic Deep Learning with Mathematical Forensic Feature Extraction, highlighting the practicality and scalability of mathematical forensics for real-world deployment.
Benefiting from recent advances in vision-language models (VLMs), numerous CLIP-based zero-shot anomaly detection (ZSAD) methods have been proposed to address the cold-start problem. Despite their impressive performance, these methods still depend on manual prompt engineering, and their coarse-grained text prompts struggle to capture the diverse patterns of anomalies, resulting in suboptimal visual-text alignment. To overcome these limitations, we propose VCF-CLIP, a visual context-driven fine-grained prompt learning framework built upon CLIP. The novelties of VCF-CLIP lie in two main aspects. First, we propose the prompt prototype learning (PPL) strategy, which learns a pair of unified prompt prototypes representing general normal and anomalous states in a loss-guided manner, thereby eliminating the need for manual prompt design. Second, we propose a lightweight prompt refinement adapter that dynamically aggregates multiscale and multilevel visual features to iteratively refine the prompt prototypes, enabling the generation of instance-specific prompts enriched with fine-grained information. We conduct extensive experiments on 14 benchmarks across industrial and medical domains, and show that VCF-CLIP outperforms existing state-of-the-art ZSAD methods.
Kaiwen Fu, Fei Qi, Chengyuan Chang et al.· IEEE Transactions on Neural...· 0 citations
Applying general-purpose vision-language models (VLMs) to crop disease diagnosis presents three critical bottlenecks: reliance on large-scale annotated data, the high computational cost of full finetuning, and existing adaptation methods designed mainly for discriminative classification without sufficient visual-linguistic interaction for generative diagnosis.
We propose BLAP, an adaptive multi-scale visual prompt fine-tuning framework built upon BLIP-2. BLAP introduces an adaptive visual prompt fusion module (APFM) with learnable prompt vectors and a gating mechanism, together with a multi-scale pyramid feature fusion module (PFM). All BLIP-2 backbone parameters are frozen, and only 0.11% of the model parameters are optimized.
On a few-shot dataset comprising 990 images from 11 crops and 33 disease categories, BLAP achieved 92.78% recognition accuracy, outperforming the BLIP-2+LoRA baseline by 21.67 percentage points. BLEU-4 and ROUGE-L scores reached 0.6507 and 0.7184, respectively, while inference latency increased by only 2.15%.
BLAP provides a lightweight solution that balances accuracy, efficiency, and interpretability for crop disease diagnosis in resource-constrained settings. The proposed dynamic prompt fusion and multiscale pyramid adaptation strategy may also be extended to parameter-efficient fine-tuning of visionlanguage models in other domain-specific applications.
Jiayi Xiong, Libin Li, Yiqing Wang et al.· Frontiers in Plant Science· 0 citations