Toward event anticipation: multimodal semantic contrastive learning for video action anticipation
Abstract
Video action anticipation aims to predict an upcoming action before it actually occurs based only on previously observed video frames, and can be viewed as a fine-grained form of event anticipation in streaming videos. Existing methods mainly rely on visual temporal modeling to infer future representations, while recent LLM-based approaches typically introduce language as an external prior or high-level predictor, making the injected semantics relatively coarse and indirect and thus unable to provide explicit fine-grained supervision for future feature generation, which is critical to anticipation. To address this issue, we propose a multimodal semantic contrastive learning framework that introduces language-derived fine-grained semantic supervision into future representation learning. Specifically, we first construct action-aware text prototypes by prompting a large language model to generate multiple detailed descriptions for each action and encoding them with a VLM text encoder, and then align the generated future features with the corresponding prototypes through contrastive learning to improve the semantic discriminability of future representations. Extensive experiments on EPICKITCHENS- 100 and THUMOS’14 demonstrate the effectiveness of the proposed method.