Serial Cycle-Consistent Video Joint Embedding Predictive Architecture with Phase-Wise Optimization

Abstract

Joint embedding predictive architectures offer a compelling alternative to pixel space reconstruction for visual self-supervised learning, especially in video. Existing video joint embedding predictive architecture methods primarily emphasize one-way masked prediction, where the model predicts masked target representations from visible context. However, the predicted latent tokens are not explicitly constrained to preserve enough information to recover a more complete semantic target. This allows shortcut solutions in which predicted tokens are sufficient for local matching while discarding global structure. In this work, we propose a serial cycle-consistent extension to video joint embedding predictive architecture. The method first performs standard forward masked prediction in latent space and then applies a backward predictor to reconstruct full target features from the predicted masked tokens. This yields a cycle-consistency regularizer that encourages structurally recoverable latent representations. To stabilize optimization, we adopt a two-stage training procedure. Phase 1 warms up the backward branch under a frozen encoder and forward predictor, and Phase 2 jointly optimizes both objectives with an exponential moving average teacher. We further evaluate the learned representations with frozen probe protocols on downstream video classification and action anticipation tasks, using only the pretrained encoder as the transferred backbone. Our experiments show that the proposed architecture converges stably without representation collapse and learns robust spatiotemporal representations that better preserve global semantic structure. More broadly, by improving the predictive usefulness and recoverability of latent video representations, the proposed framework highlights the potential of JEPA-style world models for action understanding, anticipation, and future embodied robotic perception and decision making.