TL;DR
Experimental results in various task scenarios show that the proposed framework consistently improves overall task success rates compared with unimodal settings with different LLMs and achieves a higher success rate compared to using only visual or force data.
VLA-SP (Vision-Language-Action via Symbolic Planning), a two-stage Embodied Vision-Language-Action framework, enabling fully automated robotic execution from speech and vision inputs is proposed, demonstrating the strong interpretability, executability, and cross-platform applicability of the framework.
A closed-loop framework for planning and evaluation of a vision-language model-based robotic manipulation agent operating in tabletop object rearrangement and manipulation tasks and demonstrates the potential of closed-loop vision-language planning for human-centered robotic manipulation.
Evidence Acquisition and Feasibility Gating (EAFG) is proposed, a framework that acquires visual evidence through VLM-generated exploratory subgoals and TAMP-based execution and applies a feasibility gate to decide whether to proceed with task planning, acquire further evidence, or halt.
The remarkable progress of Multimodal Large Language Models (MLLMs) has attracted increasing attention to extending them to physical entities like legged robots and robotic arms. This typically requires MLLMs to not only grasp multimodal understanding abilities, but also integrate visual-spatial reasoning and physical interaction capabilities. Nevertheless, existing methods struggle to jointly accommodate these capabilities within a shared multimodal model, due to the substantial differences between vision-language reasoning and low-level physical control. In this paper, we present Visual Embodied Brain-1.5 (VeBrain-1.5), a task-level unified framework that connects multimodal perception and spatial reasoning with robot control through a shared MLLM-compatible decision interface. VeBrain-1.5 reformulates robotic control into common text-based MLLM tasks in the 2D visual space, thereby aligning the supervision format and prediction interface of multimodal understanding, spatial reasoning, and robot control. A modular robotic adapter subsequently translates these high-level textual decisions into embodiment-specific motion commands and executes them in a closed loop. From the data perspective, we further introduce VeBrain-1M, a high-quality instruction dataset encompassing various capabilities of VeBrain-1.5. In VeBrain-1M, we take hundreds of hours to collect, curate and annotate the data, and adopt multimodal chain-of-thought (CoT) to mix the different capabilities into a single conversation. Extensive experiments on 13 multimodal benchmarks, 7 spatial intelligence benchmarks and 4 simulation and real-world robotic benchmarks demonstrate the superior performance of VeBrain-1.5 over existing MLLMs like Qwen2.5-VL. To further eliminate the task ambiguity and enhance the reasoning capability, we adopt an offline RL policy learning with verifiable reward to increase the task success rate. When deployed to two robotic entities including legged robots and robotic arms, VeBrain-1.5 shows strong adaptability, flexibility, and compositional capabilities compared to existing methods.
This study comparatively examines the task-oriented grasping problem, which is of critical importance in robotic manipulation, through modern Vision-Language Models. Within the scope of the study, the performance rates of the GraspMolmo model, specifically trained for robotic tasks, and Gemini ER-1.5, a general-purpose multimodal AI, were analyzed. The evaluation process was conducted using the TaskGrasp-Image dataset, which encompasses a wide range of objects and tasks, through natural language commands and RGB-D images. The accuracy of the grasping coordinates generated by the models was systematically assessed across varying tolerance thresholds, revealing that both models achieved high accuracy rates. The robotics-specialized model demonstrated a notable advantage over the general-purpose model, particularly under strict tolerance conditions. Error analysis showed that the majority of failed predictions targeted functionally incorrect regions of the object rather than falling outside it entirely, indicating that semantic reasoning rather than geometric localization constitutes the primary challenge.
This work proposes APIVOT, a VLM-based planner that adaptively interleaves language and visual thoughts for long-horizon planning that outperforms general-purpose VLMs and prior planning frameworks, achieving the largest gains in spatially constrained settings.
