SimVFMP: Simulation-Verified Foundation Model Planning for Robotic Manipulation

Abstract

Robotic manipulation demands generalist systems that adapt to diverse tasks without task-specific retraining. Typically, these roles are handled by foundation models for high-level planning or simulation for motion validation, but existing approaches fail to unify semantic reasoning with physical feasibility checks before deployment. In this paper, we present SimVFMP (Simulation-Verified Foundation Model Planning), a framework that integrates multimodal large language models with a high-fidelity digital twin in Isaac Sim for iterative, risk-free plan refinement. We show that SimVFMP improves task completion rates and sorting accuracy over baselines that lack simulation-based verification, while reducing unsafe physical executions. We instantiate our method through a hierarchical agent architecture featuring reflective planning, subtask success detection, and normal-based grasp synthesis. Our experiments, conducted on a bimanual robotic cell across pick-and-place, sorting, and stacking tasks, demonstrate a 20% overall success rate and 77.78% sorting ratio, outperforming the direct execution on the real cell (0% and 64%, respectively).