Skip to content

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

Sep 2026 · arXiv.org · Vol abs/2609.33419 · 0 citations · 52 references
Computer Science

TL;DR

TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2.

Abstract

Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective, data exposure, schedule, scale, and decoder capacity can all vary at once. This makes it hard to identify which choices yield motion-prioritized representations, whose gains concentrate on frame-to-frame change while retaining useful appearance. We address this with a matched $4 \times 6 = 24$ architecture-objective study at roughly 170M ~ 190M encoder scale on $\sim$1.7M OpenVid and Moments-in-Time v2 clips for 8 epochs, and propose TT-VidT. TT-VidT combines a DINOv3-initialized ViT-B/16 per-frame spatial path with a compact Temporal Transfer Layer, trained by Diff Compression to reconstruct target frames from a first-frame appearance anchor and frame-specific motion tokens. The sweep shows that TT3D with Diff Compression, not either component alone, enters the strongest motion-sensitive regime, and decoder ablations favor a compact video-pretrained decoder. In final comparison, TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2. HMDB51, IARD, and EPIC-Kitchens bound the claim.

View source

Similar papers

Preprint Sep 2026

DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding

DSE-VTG is proposed, a \underline{D}ual-\underline{S}ide \underline{E}nhancement framework that addresses both temporal dynamics and task-specific training without any task-specific training, and achieves state-of-the-art performance among training-free methods.

Zhuo Cao, Bing-Qing Zhang, Sen Wang et al. · 0 citations
Preprint Aug 2026

Semantic Slots for Video Object-Centric Learning

Video Object-Centric Learning (OCL) has traditionally focused on refining the encoder architecture to ensure temporal consistency. In this paper, we argue that the primary bottleneck lies in the decoder. We show that traditional decoders force slots to be spatially anchored, hindering their ability to adapt to motion....

K. Sabri, Guillaume-Alexandre Bilodeau, Nicolas Saunier et al. · 0 citations
Preprint Aug 2026

LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding

LongVU-TTT is introduced, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM, and is stronger than attention- and fixed-state recurrent resamplers across three benchmarks.

Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase et al. · 0 citations
Conference Open access Oct 2026

SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection

To adapt billion-parameter Vision Transformers for long-video understanding, recent methods freeze the backbone and train lightweight convolutional modules. While effective for parameter-efficient training, existing adapters do not reduce inference-time computation, leaving scalability with respect to video length larg...

Ricardo Pizarro, Roberto Valle, J. M. Buenaposada et al. · 0 citations

Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

This work instantiates VDN on MiniMax H3, which combines local Softmax attention with bidirectional linear memory for long-range video context and introduces Video Delta Attention (VDA), which updates memory once per frame by jointly incorporating its spatial tokens.

Hao-Cheng Xi, Yiming Xie, He-Xue Zhao et al. · 4 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.