Skip to content

Latent evolving World Action Model

Sep 2026 · arXiv.org · Vol abs/2609.27455 · 1 citation · 64 references
Computer Science

TL;DR

This paper proposes LeWAM, which conditions action generation on JEPA embeddings and models environment evolution by predicting future embeddings in the same space, without relying on a video diffusion backbone, and introduces Demonstration-Guided DPO (DemoDPO), an offline preference refinement stage that derives preference supervision directly from demonstrations.

Abstract

World Action Models (WAMs) jointly model action generation and environment dynamics and are mostly built on pretrained Video Diffusion Models (VDMs). In VDM-based WAMs, observations are first encoded by a VAE, and the resulting compressed latents are then processed by large video diffusion backbones to extract effective features for action generation. However, this paradigm ties WAM performance and training cost to large-scale video generation pretraining, limiting WAM efficiency and scalability. In this paper, we theoretically and empirically investigate how visual representations affect action generation in WAMs. Our results show that predictive embeddings from Joint-Embedding Predictive Architecture (JEPA) encoders better support action generation than compressed VAE latents, with I-JEPA performing best in our encoder comparison. Based on these findings, we propose LeWAM, which conditions action generation on JEPA embeddings and models environment evolution by predicting future embeddings in the same space, without relying on a video diffusion backbone. We further find that imitation learning matches demonstrated actions but does not distinguish better actions from worse ones, even though small action deviations can greatly affect task success. To address this limitation without additional environment interaction or the human oversight required for resets and safety, we introduce Demonstration-Guided DPO (DemoDPO), an offline preference refinement stage that derives preference supervision directly from demonstrations. With only 0.4B trainable parameters, LeWAM achieves an average success rate of 92.28\% on RoboTwin 2.0, comparable to that of state-of-the-art VLAs and WAMs, and maintains practical effectiveness on real-world manipulation tasks.

View source

Similar papers

Sep 2026

Modality-Autoregressive World-Action Models

ModAR is introduced, the first WAM to autoregressively denoise multiple future modalities before predicting actions, and it is found that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales.

Adam Hung, B. P. Duisterhof, D. Ramanan et al. · 4 citations
#machine learning Preprint Nov 2026

La-JEPA: Compact Robot Policies with Latent World Modeling

It is shown that JEPA features outperform video VAE latents, most clearly out of distribution, and that training-time future prediction choices significantly impact the policy even when future prediction is skipped at inference, suggesting that robot foundation models may benefit from architectures designed for control...

Mateo Clémente, George Z. Xue, Amin Abyaneh et al. · 0 citations
#machine learning Preprint Oct 2026

SUAVE: Unified Video-Action Models via Masked Diffusion

A single SUAVE model predicts long-horizon video and acts as a policy, competitive with dedicated world models and specialized action policies on static and dynamic manipulation tasks and shows that masked diffusion is a practical and versatile foundation for unified video-action modeling.

Rhythm Syed, Jean-Pierre Mercat, Sedrick Scott Keh et al. · 0 citations
Preprint Oct 2026

Self-Distilling Future Video Representations for World Action Models

Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step, and substantially improves human-to-robot skill transfer.

Huiwon Jang, Suhyeok Jang, Hyeon-Joo Kang et al. · 0 citations
Preprint Oct 2026

LaLa-World: Scaling Latent World Models with Natural Language Actions

This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this...

Jun-Ha Song, Geonmo Gu, Jeongeun Park et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.