Multi-Model Large Language Models for Human-Robot Interaction

Abstract

Human–robot interaction (HRI) becomes challenging when robots must interpret human instructions expressed through multiple forms of communication. People often combine speech, pointing gestures, visual cues, and contextual information, and these signals may be incomplete, ambiguous, or inconsistent. Multimodal large language models (MLLMs) can combine such inputs to support more natural interaction, but unreliable interpretations and hallucinated information may still lead to incorrect robotic decisions. This paper proposes a reliability-focused multimodal framework for MLLM-based HRI that integrates voice, pointing gestures, visual perception, and contextual cues for human intention and object understanding. The framework processes information from each modality and combines the resulting evidence within the MLLM reasoning stage. It then examines whether the modalities support the same interpretation, identifies conflicts, estimates uncertainty, and verifies the predicted interpretation against available visual and sensor evidence. When the evidence provides sufficient support for a safe decision, the system selects the appropriate response. When the information is ambiguous, contradictory, or poorly supported, the system asks the user for clarification instead of making an unsupported assumption. Contextual and emotional cues are used as additional evidence to improve interpretation rather than as definitive measures of a person's internal state. The proposed framework is evaluated through controlled HRI scenarios involving ambiguous instructions, conflicting speech and gestures, contextual changes, and hallucination-related cases. Evaluation considers intention accuracy, conflict-detection accuracy, hallucination rate, clarification effectiveness, response time, task-completion rate, and unsafe-action rate. By combining multimodal reasoning with uncertainty estimation, verification, and clarification, the proposed framework aims to improve the reliability, safety, and trustworthiness of MLLM-based human–robot interaction.