Interleaved LLM and Motion Planning for Scalable Multi-Object Collection in Large Scene Graphs

Abstract

Household robots have been a longstanding research topic, but they still lack human-like intelligence, particularly in manipulating open-set objects and navigating large environments efficiently and accurately. To push this boundary, we consider a scalable multi-object collection problem in large scene graphs, where the robot needs to pick up and place multiple objects across multiple locations in a long mission of multiple human commands. This problem is extremely challenging since it requires long-horizon planning in a vast action-state space under high uncertainties. To this end, we propose a novel interleaved LLM and motion planning algorithm Inter-LLM. By designing a multimodal action cost similarity function, our algorithm can both reflect the history and look into the future to keep optimizing mission performance on the fly. Extensive simulation experiments of three scenes demonstrate that compared with related works, our algorithm improves the overall mission performance by 31.2% in terms of fulfilling human commands, maximizing success rates, and minimizing execution costs. Full videos https://sites.google.com/view/inter-llm