Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.
RMBench is introduced, a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity, enabling systematic evaluation of policy memory capabilities and Mem-0, a modular manipulation policy with explicit memory components designed to support controlled ablation studies.
Tian-Xing Chen, Yu-Ran Wang, Ming-Lei Li et al.· arXiv.org· 62 citations· ⚡23
This work characterize models from this family of methods as JEPA-WMs and investigates the technical choices that make algorithms from this class work, and proposes a model that outperforms two established baselines, DINO-WM and V-JEPA-2-AC, in both navigation and manipulation tasks.
Basile Terver, Tsung-Yen Yang, Jean Ponce et al.· Trans. Mach. Learn. Res.· 49 citations· ⚡7
This work introduces StarVLA-$\alpha, a simple yet strong baseline designed to study VLA design choices under controlled conditions that deliberately minimizes architectural and pipeline complexity to reduce experimental confounders and enable systematic analysis.
Jinhui Ye, Ning Gao, Sen-Qiao Yang et al.· arXiv.org· 49 citations· ⚡11
A novel vision-language model (VLM)-based navigation framework that addresses challenges by adaptively retrieving task-relevant cues from a global memory module and integrating them with the agent's egocentric observations and enhances spatial reasoning and decision-making in long-horizon tasks.
The Very Big Video Reasoning (VBVR) Dataset is introduced, an unprecedentedly large-scale resource spanning 200 curated reasoning tasks following a principled taxonomy and over one million video clips, approximately three orders of magnitude larger than existing datasets.
Maijunxian Wang, Rui-Si Wang, Ju-Yi Lin et al.· arXiv.org· 28 citations· ⚡5
Show-Harness, an Embodied Harness that enables VLMs to"play" robots through a compact semantic interface linking intent to action, and develops GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to"play"robots across embodime...
Yan-Zhe Chen, Ze-Chen Bai, Zhi-Jun Cao et al.· 26 citations· ⚡2
Robots are getting smarter, but how can their hardware match that growth? New Microsoft Research findings show that moving AI inference beyond the robot can improve task success, boost efficiency, and support more advanced physical AI workloads. The post Offloaded inference for real-world physical AI robotics appeared first on Microsoft Research.
