Speech-to-Trajectory: An LLM-Enabled Human-Robot Collaborative Framework for Generating New Assembly Tasks From Verbal Guidance

Abstract

Human-Robot Collaboration has enhanced manufacturing efficiency and productivity by integrating the precision and repeatability of robots with the adaptability of human operators. The growing demand for customized products frequently requires human operators to manually reprogram robot task plans. This process demands specialized skills such as coding and robotic programming, making it costly and inaccessible to non-expert human operators. In this study, we propose a Large Language Model-enabled Human-Robot Collaboration framework that facilitates human-robot interaction through natural language. Using the proposed framework, a human operator can teach the robot to assemble new products via verbal guidance, while the robot learns and stores the manipulation sequences for future reuse. In addition, a motion trajectory simplifier is integrated into the framework to refine assembly motion for reuse, further improving manufacturing efficiency. The proposed framework is evaluated on the NIST Assembly Task Board, demonstrating a promising direction for Human-Robot Collaboration in manufacturing.