A multimodal Graph-based Fusion (MGF) framework that learns visually grounded semantic representations to enhance FSOR performance and achieves superior open-set recognition and competitive closed-set classification performance.
Abstract
Few-shot open-set recognition (FSOR) presents unique challenges due to the limited labeled samples and the presence of unknown classes during inference. While recent few-shot learning methods leverage class-level textual information to enhance feature discrimination, they often overlook image-grounded descriptions that provide more fine-grained and instance-specific semantics. Moreover, many of these approaches require prior knowledge of the class name from labeled samples during inference, which may be impractical in open-world scenarios. In this study, we propose a multimodal Graph-based Fusion (MGF) framework that learns visually grounded semantic representations to enhance FSOR performance. MGF leverages image descriptions generated by a vision-language model as the textual supervision to guide the learning of semantic features from images. A graph convolutional network is then used to fuse semantic and visual features, enabling effective intra-class information propagation and improving discrimination between known and unknown classes. We jointly optimize a contrastive and a semantic alignment loss to promote intra-class compactness and inter-class separability. Extensive experiments on several few-shot learning benchmarks demonstrate that MGF achieves superior open-set recognition and competitive closed-set classification performance.
A novel framework that mines target knowledge using the strong reasoning capacity of Multimodal Large Language Models (MLLMs) and employs it to enhance FSS, which shows promising results and largely surpasses existing methods.
Few-shot image classification remains difficult because a model must identify novel classes from only one or a few labeled examples while preserving discriminative local information. Metric-learning methods based on Earth Mover’s Distance (EMD) improve local correspondence by representing an image as a set of regional...
Huie Zhang, Mary Jane C. Samontet· International journal of com...· 0 citations
Extensive experiments on three challenging fine-grained benchmarks demonstrate that MASFNet consistently outperforms state-of-the-art methods in both 5-way 1-shot and 5-way 5-shot settings.
Jinyu Wang, Bing-Xin Xu, Weiguo Pan et al.· International Conference on...· 0 citations
The Dual Adaptive Visual-Semantic Prompt Collaboration Network (VSPCN+), a novel framework that utilizes prompt-tuning for effective feature adaptation, is proposed, which introduces a dual-prompt mechanism comprising both visual and semantic prompts.
Huajie Jiang, Zheng-Xian Li, Yuankai Qi et al.· International Journal of Com...· 0 citations
G2D is proposed, a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image and transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning.
Zehua Hao, Fang Liu, Qinliang Wang et al.· 0 citations
A novel self-supervised approach that leverages generative uncertainty from text-to-video (T2V) diffusion models to learn semantic similarity without human annotations to identify stable semantic features within existing pretrained representations, such as VideoMAE and V-JEPA.
Enrico Pallotta, Sina Raoufi, Lars Doorenbos et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.