A cross-modal temporal alignment framework that combines a multi-scale temporal convolutional encoder with capsule-based dynamic routing, jointly optimizing temporal boundary prediction, cross-modal semantic alignment, and capsule diversity is proposed.
Abstract
Action grounding in videos aims to identify precise temporal boundaries in untrimmed videos that correspond to natural language queries, a capability essential for vision–language–action systems and embodied intelligence. Existing methods suffer from two limitations: treating all temporal scales uniformly through global attention, and assuming simplistic one-to-one correspondences between video frames and text tokens. To address these issues, we propose a cross-modal temporal alignment framework that combines a multi-scale temporal convolutional encoder with capsule-based dynamic routing. The multi-scale encoder applies parallel 1D convolutions with varying kernel sizes to capture hierarchical motion patterns at $O(T)$ complexity, enabling efficient modeling of both instantaneous transitions and extended action sequences. The capsule-based routing mechanism iteratively refines video–text alignments through agreement-based coefficient updates, producing sparse and structured many-to-many correspondences between video frames and text tokens. These two components are unified through a multi-task objective jointly optimizing temporal boundary prediction, cross-modal semantic alignment, and capsule diversity. Experiments on ActivityNet Captions demonstrate that our framework achieves 42.9% Recall@0.5 and 41.1% mean IoU while maintaining linear-time efficiency for long-video processing.
Video-text temporal localization requires precise alignment between natural language queries and corresponding video segments, a fundamental challenge in multimodal understanding. We present a novel framework that addresses two critical limitations of existing methods: inadequate modeling of hierarchical temporal structure and inability to handle complex many-to-many correspondences between modalities. Our approach introduces a multi-scale temporal convolutional encoder that captures motion patterns across different temporal granularities - from instantaneous frame transitions to extended action sequences. We further propose a capsule-based dynamic routing mechanism that iteratively refines segment-query associations through structured agreement updates, enabling flexible modeling of non-monotonic alignments. These components are unified through a multi-task learning objective that jointly optimizes temporal boundary regression, cross-modal semantic alignment, and capsule diversity. Extensive experiments on ActivityNet Captions demonstrate significant improvements, achieving 42.9% Recall@0.5 and 41.1% mean IoU, surpassing strong transformer-based baselines while maintaining computational efficiency. Our results validate that combining hierarchical temporal modeling with structured semantic routing provides an effective solution for fine-grained video-language understanding.
Gengtian Shi, Jinze Yu, Chenhao Wu et al.· 0 citations
This work introduces ActionLMM, a memory-augmented vision-language model for long-video action summarization that aligns visual and motion modalities through joint representation learning and leverages a novel dual-memory mechanism to retain both local motion details and global temporal structure.
Ruirui Li, Dari Abdullah Alrwoaily, Turgut Sofuyev et al.· International Conference on...· 0 citations
We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances while overcoming high annotation costs for training via new target actions and pretraining using large-scale action scenery datasets. Specifically, our approach, termed Skeleton-Language feature Pooling Switching, introduces a weakly-supervised vision-language pretraining mechanism. This mechanism transitions pooling kernels from pretraining, which aggregates skeleton features at the video level and aligns them with each video's known action text embeddings, to the inference phase that computes instance-level features without training via target actions. Furthermore, we propose Scene-Mixed Discriminative Contrastive Learning to distinguish actions at the instance level within the combined scene through the MIL framework. Our experiments on four public spatio-temporal action localization and classification datasets demonstrate that the proposed method effectively addresses annotation limitations.
Koshiro Nagano, Fumiaki Sato, Ryo Hachiuma et al.· 0 citations
Video-text models adapted from image-text architectures (e.g., CLIP) frequently exhibit temporal blindness, the inability to perceive fundamental cues like order, direction, and motion dynamics. Standard datasets mask this limitation by enabling models to exploit static spatial shortcuts. To systematically evaluate this, we introduce XTE-Bench, a diagnostic probe revealing that even large-scale video-language models struggle with basic temporal reasoning, indicating that parameter scaling alone is insufficient to resolve this flaw. To address this, we propose Cross-Modal Temporal Edits (XTE), a self-supervised framework that injects precise temporal supervision. By performing synchronized video-text transformations, XTE generates hard temporal negatives without manual annotation. We instantiate this with ViTAL-X, a lightweight model that equips frozen image-text backbones with temporal awareness while preserving their foundational spatial knowledge. Across six temporal benchmarks, ViTAL-X achieves state-of-the-art performance. Utilizing only 0.4B parameters and 1M training clips, ViTAL-X outperforms 7B-parameter models and surpasses baselines trained on 600x more data. These results demonstrate that targeted, high-quality temporal alignment provides a highly efficient alternative to pure scaling.
V. SethuramanT., Savya Khosla, O. Susladkar et al.· 0 citations
Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs. We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task: long-video labels often rely on brittle one-pass annotation, while reinforcement-learning rewards either fail to distinguish non-overlapping predictions or require fragile segment matching. TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. TimeLens2-93K constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. Our temporal Wasserstein reward computes exact one-dimensional \(W_1\) between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation; temporal IoU complements it with precise-overlap feedback. Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.
Yuhan Zhu, Changlian Ma, Xiangyun Zeng et al.· 2 citations
Temporal action detection is a crucial task in the field of video understanding, aiming to localize and recognize the category of actions in untrimmed videos along with their precise start and end times. Despite the success achieved by existing methods, two significant challenges remain: (1) difficulty in modeling long-range dependencies between video segments, leading to inaccurate localization of complex action boundaries; and (2) over-reliance on temporal dimension modeling, with insufficient exploration of the representational capabilities within the channel dimension, which limits model performance. To address these challenges, we propose a Multi-scale Temporal-Channel Collaborative (MTCC) detection framework. First, we construct a Long-Range Dependency Enhancement (LRDE) module based on the Mamba architecture with state space models (SSMs), which efficiently capture long-range temporal dependencies in video sequences. Second, leveraging the advantages of multi-scale CNNs, a Multi-Scale Boundary Awareness (MSBA) module is designed to extract local features and enhance the model's sensitivity to action boundaries. Finally, a Cross-Channel Information Fusion (CCIF) module is designed to extract features from the channel dimension of video. To better integrate features at different scales, a Multi-scale Detection Head (MDH) is employed to dynamically fuse the feature pyramid. Our systematic evaluations on the benchmark datasets THUMOS14 and ActivityNet-1.3 yield impressive results, demonstrating the effectiveness of our proposed method.
Yibo Zhao, Wen Zhang, Chunjie Ma et al.· ACM Transactions on Multimed...· 0 citations