Skip to content
Preprint

Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognition

Oct 2026 · 0 citations · 51 references
Computer Science

TL;DR

P2CLIP (Prompt-guided CLIP), which aims to bridge the gap by exploring the impact of prompts across three aspects: category, video, and frame, and proposes a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a better construction of the joint visual-textual feature space.

Abstract

Vision-language models (VLMs) like CLIP have shown great success in image tasks, however, applying CLIP to open-vocabulary video recognition poses challenges. Due to the limited text information, directly fine-tuning CLIP on video recognition datasets may lead to overfitting and hinder its generalizability. A prevalent method is to enrich text information by generating prompts using Large Language Models (LLMs). However, limited efforts have been made to explore how to generate higher-quality prompts that can contribute to classify unseen categories, and how to further enhance vision-text alignment based on these prompts. In this paper, we propose P2CLIP (Prompt-guided CLIP), which aims to bridge this gap by exploring the impact of prompts across three aspects: category, video, and frame. At the category level, we argue that LLM-generated prompts under simple instructions may suffer from semantic overgeneralization, which can blur the category boundaries and hinder the performance of the model. To address this issue, we design Generic-Discriminative Prompt using LLMs to obtain discriminative category embeddings with rich semantic information. At the video and frame levels, we first use VLM and LLM to generate frame and video descriptions for enhancing vision-text alignment. Then we propose a Knowledge-Guided Temporal Aggregation mechanism, which leverages the expert knowledge derived from LLMs to guide the temporal modeling of videos and encourage a better construction of the joint visual-textual feature space. Extensive experiments across various benchmarks, including zero-shot and few-shot settings, demonstrate the effectiveness of our method.

View source

Similar papers

Sep 2026

Making Large Vision–Language Models Better Few-Shot Learners

Few-shot classification (FSC) aims to emulate the human ability to rapidly learn new concepts from a handful of examples. Large Vision-Language Models (LVLMs), with their rich prior knowledge and powerful visio-linguistic understanding capabilities, are emerging as a highly promising paradigm for FSC. This paper invest...

Chuan-Yi Zhang, Fan Liu, Yi Xu et al. · 0 citations

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

ProCAP is proposed, a probabilistic cross-attentive prompt learning framework that improves cross-modal interaction and training stability without updating any CLIP weights: it learns both visual and textual prompt tokens and links them through stacked bidirectional multi-head cross-attention so the two branches refine...

H. Abbas, Fatemeh Daneshfar, M. Abdar · 0 citations

Query-aligned video frame selection for long video understanding

This work introduces a frame-selection method designed specifically for multiple-choice questions, and is the first to directly utilize answer choices as inference-time cues for selecting frames relevant to answering a question.

Md. Safayet Islam, Dilip Sarkar, Liang-Yu Liang · 1 citation
Book Open access Nov 2026

DraftMind: Efficient Video Understanding in Vision-Language Models via Chain of Draft

Large vision-language models (LVLMs) such as Qwen-VL series have demonstrated strong capabilities in visual understanding and reasoning, particularly for static images. However, extending them to video reasoning remains computationally expensive, often incurring substantial latency and token usage under conventional Ch...

Zhao-Yang Wei, Wen-Chao Ding, Chen-Hui Qiang et al. · 0 citations

Advancing Video-Text Pretraining with Multi-View Captions

This work proposes a large-scale multimodal large language model-based supervision generation framework that improves supervision diversity, fidelity, and semantic coverage, and introduces a granularity-aware text representation with separate CLS tokens for summary and detailed views.

Fida Mohammad Thoker, Renaud Vandeghen, Karen Sánchez et al. · 0 citations
Preprint Oct 2026

SIMPLE: Semantic Structure-Aligned Prompt Learning for Open-Vocabulary Audio-Visual Event Localization

This paper proposes SIMPLE, a semantic structure-aligned prompt learning framework that aligns cross-category semantic topology between learnable and hand-crafted prompts across asynchronous audio and visual streams and achieves state-of-the-art performance on the seen and unseen splits.

Han-Yu Xuan, Hao-Ran Xu, Ming-Kun Yin et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.