Skip to content

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

Sep 2026 · arXiv.org · Vol abs/2609.38078 · 1 citation · 31 references
Computer Science

TL;DR

MotorMind is introduced, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates, and shows that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchronous execution harness, can perform effective zero-shot robotic manipulation.

Abstract

Vision-language-action (VLA) models have advanced robotic manipulation, but their zero-shot generalization in new tasks and environments remains limited, and their reliance on specialized training keeps them from benefiting directly from rapidly advancing general-purpose vision-language models (VLMs). In parallel, recent agentic robotic systems leverage VLMs for high-level reasoning or coding agents for robot control, but often depend on extensive external models and tools, introducing additional complexity and cost. This motivates us to ask: Can a general-purpose VLM itself operate a robot more like the human teleoperator by reasoning directly from observations, issuing actions, and continuously adapting to execution feedback, without relying on external models such as learned action experts, coding agents or grounding tools like SAM3? In this work, we introduce MotorMind, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates. Without task-specific policy training, coding agents, or additional grounding tools such as SAM3, MotorMind achieves 66.7% success on the base LIBERO-PRO suites and 53.8% under perturbations, compared with at most 13.3% and 19.2%, respectively, for the prior zero-shot methods we evaluate. The same interface reaches 95% average success on a real xArm6 robot across direct manipulation and human-perturbation settings. Replacing the backbone with a stronger VLM further improves performance, while the remaining failures - primarily due to visual grounding, embodied reasoning, and action knowledge - decrease as VLM capability improves. These results show that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchronous execution harness, can perform effective zero-shot robotic manipulation.

View source

Similar papers

GT-VLA: Target-Conditioned Trace Guidance for Generalizable Robotic Manipulation

Vision-Language-Action (VLA) models have shown strong performance on robotic manipulation, but they often struggle to generalize to unseen tasks, configurations, and long-horizon settings. A key challenge is that VLAs overfit to training scenes and fail to follow novel language instructions. Off-the-shelf vision-langua...

Ning-Han Zhong, Jing-Chen Peng, Sriram Vishwanath · 0 citations
Preprint Sep 2026

Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation

Robo-Harness K1 is introduced, a robot-use agent (RUA) framework that exposes perception as tools that makes 3D geometry accessible without changing the VLM architecture or training a depth encoder, and suggests that perception-augmented RUAs offer a promising route to sample-efficient, generalizable robotic policies t...

Ze-Xi Li, Ye-Hang Zhang, Wen-Qian Li et al. · 6 citations · ⚡2
Preprint Aug 2026

TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation

TemporalFlow-VLA provides a compact, physically grounded interface for exploiting ordered execution history without explicit motion estimation or geometric processing at deployment, and shows its clearest advantage over prior methods on longer-horizon, multi-stage manipulation.

Jia-Rui Yang, Ye-Hao Lu, Yu-Ning Su et al. · 2 citations
#small language model Preprint Sep 2026

SkipVLA: Skipping VLA Steps with Classical Planning for Fast Robot Manipulation

SkipVLA is a hybrid policy that combines a pretrained VLA with a classical motion planner, using the planner for free-space motion and querying the VLA only for contact-rich skills such as grasping and placing, demonstrating up to 2.5x faster task completion and significantly lower energy consumption.

Kaivalya Agrawal, Md Ashiqur Rahman, Raymond A. Yeh et al. · 0 citations

HIL-UMI: Bringing Human-in-the-Loop Post-Training of Vision-Language-Action Models to Universal Manipulation Interface

Large-scale vision-language-action (VLA) models provide powerful priors for robot manipulation, yet adapting them to a specific deployment remains challenging. Supervised fine-tuning (SFT) on task-specific demonstrations provides a step toward deployment, but faces two persistent limitations: static data provide limite...

Zimu Han, Yi-Ming Zeng, Ji-Yao Zhang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.