Unsupervised Interpretable Human Intent Discovery in Teleoperation via Bayesian Nonparametric Learning and Multi-Modal LLMs

Abstract

Human intent recognition is crucial for effective robotic teleoperation, yet existing methods rely on a predefined set of intentions, a requirement that is manually intractable in complex open-world scenarios. We propose an unsupervised framework that integrates Bayesian nonparametric learning with multi-modal Large Language Models (LLMs) to autonomously discover and interpret human intentions. We first employ a hierarchical Dirichlet process hidden semi-Markov model (HDP-HSMM) to adaptively segment kinematic sequences into motion primitives, reducing the load on downstream models. Each segment is represented by multi-modal features extracted from front-facing video, robot kinematics, and eye fixations. We then develop a reasoning architecture based on zero-shot Chain-of-Thought (CoT) prompting, where a Vision-Language Model (VLM) translates each segment into a semantic description, and an LLM clusters these descriptions to reveal intentions and generate human-readable interpretations. Experimental results on real-world data show that our method can uncover meaningful intentions highly consistent with expert judgments, outperforming baselines in intra-intent consistency by achieving the lowest semantic outlier rate of 8.5%.