Large Language Model-Driven Symbolic Planning for Long-Horizon Robotic Manipulation Tasks

Abstract

Cross-modal foundation models are increasingly used for robotic task understanding and planning. However, connecting multimodal observations and natural language instructions to symbolic planners and executable robot actions remains a key challenge in embodied AI. In addressing this challenge, existing methods often rely on structured inputs and lack robustness to language ambiguity, with limited real-world validation. We propose VLA-SP (Vision-Language-Action via Symbolic Planning), a two-stage Embodied Vision-Language-Action framework. VLA-SP consists of a Multi-modal Information Processing Module and a PDDL-based Vision-Language-Action Module, enabling fully automated robotic execution from speech and vision inputs. First, our framework detects task-relevant objects using YOLO and transcribes voice instructions with Faster-Whisper. After obtaining the visual observations and language instructions, the LLM fuses these multimodal inputs and converts them into a planner-verifiable problem.pddl instance under a reusable domain.pddl. Then, Fast Downward validates the generated instance and, upon successful verification, produces a multi-step symbolic plan. If any issues are detected during verification, planner feedback is used for traceable failure diagnosis and iterative self-correction. Once a valid symbolic plan is obtained, the symbolic actions are further grounded into 6-DoF poses and executed through predefined robot skills. Finally, we conduct extensive experiments across seven planning domains in simulation and six real-world manipulation tasks on two robotic platforms, X-Arm and Franka. The experimental results demonstrate that VLA-SP enables reliable execution with high success rates across diverse manipulation tasks. In particular, the complete framework achieves an average success rate of 82% across the six real-world tasks. Results from both simulated and real-world experiments fur-ther demonstrate the strong interpretability, executability, and cross-platform applicability of our framework. Further details are available here.