Aug 2026· Frontiers in Robotics and AI· Vol 13· 0 citations· 61 references
Medicine
TL;DR
A closed-loop framework for planning and evaluation of a vision-language model-based robotic manipulation agent operating in tabletop object rearrangement and manipulation tasks and demonstrates the potential of closed-loop vision-language planning for human-centered robotic manipulation.
Abstract
The rapid advancement of Large Language Models (LLMs) has led to significant progress in Artificial Intelligence (AI), ushering in a new era of human-centered AI (HAI). Intelligent agents powered by LLMs provide new opportunities for realizing HAI. However, existing LLM-based embodied agents often lack online planning capabilities and may generate actions involving objects that are not present in the current environment. In this paper, we propose a closed-loop framework for planning and evaluation of a vision-language model-based robotic manipulation agent operating in tabletop object rearrangement and manipulation tasks. These tasks require a robot to interpret high-level natural language commands and perform multi-step actions such as stacking objects, sorting objects by category or attribute, placing objects into target containers, and organizing mixed desktop scenes. The proposed architecture consists of a human–robot speech interaction module, a vision–language agent module (including a planner, translator, and evaluator), and an action execution module. Within this framework, the vision–language planner decomposes high-level instructions into executable task steps via Chain-of-Objects Reasoning, enabling a deeper understanding of the visual environment, including spatial relationships and object attributes. Meanwhile, the task evaluator monitors execution outcomes and provides result-oriented feedback, allowing the system to replan when errors occur. Experimental results show that, compared with baseline methods, the proposed agent improves the average task success rate by approximately 20% in both simulation and real-world environments, significantly enhancing its ability to execute complex natural language instructions. This work demonstrates the potential of closed-loop vision-language planning for human-centered robotic manipulation. Code: https://github.com/subforpaper/PFEA.
Experimental results in various task scenarios show that the proposed framework consistently improves overall task success rates compared with unimodal settings with different LLMs and achieves a higher success rate compared to using only visual or force data.
ABot-AgentOS is presented, a general robotic Agent Operating System that sits above low-level controllers and provides a deliberative agent layer for scene-conditioned planning, context-isolated skill execution, multi-stage verification, multi-modal memory, and edge-cloud collaboration.
Jiayi Tian, Shiao Liu, Yuting Xu et al.· 0 citations
Language agents can now interact fluently with users in software, but robots still struggle to bring comparable interaction to physical tasks. Current robot-control paradigms, including vision-language-action policies and world-model-based planners, are mainly optimized for instruction execution, leaving users with little visibility into why an action is chosen and few mechanisms to redirect, correct, or teach the robot through interaction. To solve this problem, we present the World-Cognition Model (WCM), a human-centered embodied agent built on the SLAK architecture (Sensing, Logic, Action, and Knowledge) and an asynchronous runtime. SLAK separates perception, reasoning, control, and memory, while the runtime allows reasoning, dialogue, and execution to proceed concurrently. WCM further introduces a human-in-the-loop teaching mode that enables users to interactively teach the robot difficult or long-horizon tasks. Teaching episodes and autonomous task rollouts are refined into chain-of-thought supervision to continually improve the model. WCM achieves a 73.8% average success rate across nine real-world human-robot interaction tasks, including tasks held out from CoT fine-tuning and a long-horizon task learned through teaching.
A structured taxonomy is presented that organizes existing work into three complementary paradigms that represent dominant architectural tendencies in current LLM-based embodied task planning research, and compares these paradigms along dimension of accuracy, robustness, scalability, efficiency, and sim-to-real transfer.
Zhen Zhang· Applied and Computational En...· 0 citations
VLA-SP (Vision-Language-Action via Symbolic Planning), a two-stage Embodied Vision-Language-Action framework, enabling fully automated robotic execution from speech and vision inputs is proposed, demonstrating the strong interpretability, executability, and cross-platform applicability of the framework.
Hanzhuo Zhang, Jiahao Xu, Yicheng Xu et al.· Tsinghua Science and Technol...· 0 citations
Embodied artificial intelligence (Embodied AI) aims to develop agents that can perceive, reason, and act through continuous interaction with physical world. To achieve this goal, agents must not only understand visual scenes and language instructions, but also transform multimodal information into executable actions. This requirement calls for models that can integrate perception, semantic reasoning, and action generation within a unified decision‐making framework. Recent advances in large language models (LLMs) have provided important technical foundations for this integration, leading to emergence of vision‐language‐action (VLA) models as a central paradigm for embodied systems. Although existing surveys have reviewed Embodied AI from perspectives such as robotic systems, simulation environments and task settings, few have systematically analyzed action‐generation mechanisms of VLA models. This limitation makes it difficult to distinguish VLA architectures. To address this gap, we provide a comprehensive review of VLA models for Embodied AI from an action‐generation perspective. We first formulate VLA models as embodied decision‐making systems that map visual observations and language instructions to action sequences through interaction with dynamic environments. Building on this formulation, we review historical evolution of VLA models. Furthermore, we propose an action‐generation‐centered taxonomy that categorizes VLA models into three paradigms: direct policy learning, generative action modeling, and reasoning‐guided modeling. For each paradigm, we analyze representative methods, action‐generation mechanisms, advantages and limitations, thereby providing a coherent framework for comparing diverse VLA architectures. Beyond model architectures, we summarize key resources and benchmarking settings and discuss recent applications. Finally, we identify major challenges and future directions. By organizing VLA models around the central problem of action generation, we aim to provide a structured technical map for future research toward more general, reliable, and physically grounded Embodied AI.
Ning Xiong, Mingle Xu, Wei Chen et al.· Journal of Field Robotics· 0 citations