A metric interaction framework that models object-level and scene-level interactions in physical Cartesian space at a shared metric scale and improves diverse VLA and WAM baselines with a small number of additional parameters and training steps is introduced.
Abstract
Vision-Language-Action models and World-Action Models have advanced language-conditioned robotic manipulation, yet often leave metric relations among actions, objects, and scene geometry implicit. Human manipulation combines semantic understanding of task-relevant objects with spatial feedback that guides hand motion relative to objects and their surroundings. Inspired by this, we introduce a metric interaction framework that models object-level and scene-level interactions in physical Cartesian space at a shared metric scale. At the object level, Interaction-Centric Tokens (ICTs) explicitly represent end-effector pose trajectories relative to manipulated objects and are jointly denoised with actions, providing physically grounded interaction supervision. At the scene level, the Metric Action Interaction Field (MAIF) uses action and ICT queries to attend to metric scene point-cloud features and learns geometry-conditioned action corrections. Through two-stage adaptation, our framework improves diverse VLA and WAM baselines with a small number of additional parameters and training steps. Experiments demonstrate average success-rate gains of 0.80 and 3.59 percentage points on LIBERO and RoboTwin 2.0, respectively, alongside gains of 6.80 percentage points on real-world tasks and 7.45 percentage points on their out-of-distribution variants.
World action models (WAMs) aim to answer a coupled physical question: given a task instruction, what motion should the robot execute, and how will that motion change the surrounding world? Most existing WAMs build on pretrained video generators and represent world evolution through images or visual latents. Robotic int...
Rui-Xiang Wang, Yong-Yi Su, Wen-Lve Zhou et al.· 0 citations
UniWAM is introduced, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction and uncovers a log-linear scaling law of unified human-robot co-training.
Wen-Xuan Song, Jia-Yi Chen, Jing-Bo Wang et al.· 6 citations· ⚡2
A spatial Gaussian-guided hierarchical framework that uses ordered 3D Gaussian interaction regions as an explicit planning interface between vision–language reasoning and action generation and demonstrates the effectiveness of explicit 3D interaction representations for spatial reasoning and long-horizon manipulation.
Li Lin, Ming-Hao Shi, Teng-Long Wang· Applied Informatics· 0 citations
Results show that interaction-centric spectral guidance provides an effective and scalable way to transfer physical interaction knowledge from human egocentric video to robot control.
Zhi-Ming Liu, Yi-Kun Miao, Ying Chen et al.· 1 citation
MachEmbodied-U0, a unified embodied foundation model connecting understanding and generation experts through a Mixture-of-Transformers architecture, is presented, a unified embodied foundation model connecting understanding and generation experts through a Mixture-of-Transformers architecture that combines competitive...
Hao-Ran Wen, Wen-Fu Wang, Kun-Song Shi et al.· arXiv.org· 1 citation
Egocentric human data offer a path to scaling robot learning beyond costly robot demonstrations, yet the embodiment gap makes raw human trajectories a poor supervisory target for control. Our key insight is that, although low-level actions are embodiment-specific, their underlying motion intent can capture task-relevan...
Li-Han Zha, Shresth Grover, Tenny Yin et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 9, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.