Jun 2026· arXiv.org· Vol abs/2606.31158· 0 citations· 7 references
Computer Science
TL;DR
A novel framework that leverages the reasoning capabilities of Large Language Models to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats.
Abstract
The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics. Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability. This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats. Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection. These processed inputs are contextualized using prompt templates and fed into a LLM. The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions. This sequence is dispatched to an action queue for execution on a quadruped robot over ROS. The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music. This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.
Voice interaction has been actively studied in human–robot interaction (HRI) for decades, yet deploying spoken interfaces on physical mobile manipulators remains challenging because language is ambiguous, tasks are long-horizon, and robot actions must be grounded to perception and motion under real-world uncertainties. Recent large language models (LLMs) offer a practical way to interpret open-ended spoken requests, but their non-deterministic outputs and limited transparency can hinder safe and reproducible robot execution. This paper presents an LLM-based Voice-to-Action (VTA) system that converts spoken user commands into grounded robot behaviors for an indoor mobile manipulator, LeeAhn 2. The system combines speech transcription with an LLM that produces structured, skill-level action plans aligned with a predefined library of robot capabilities, including vision-based seeking, wheeled navigation, and manipulation. To improve reliability, we incorporate interface constraints that restrict generated actions to executable skills and enable recovery from common failures during execution. We evaluate the proposed system through simulation experiments and real-world trials on a representative search task, reporting component-level performance across seeking, navigation, and manipulation. The simulation experiments provide repeatable analysis under controlled conditions, while the real-world trials demonstrate practical applicability on the LeeAhn 2 mobile manipulator and reveal limitations such as latency and occasional plan/execution failures. Our results suggest that LLM-grounded spoken interfaces can reduce operator burden and improve accessibility for indoor service robots.
Kisu Ok, Geunyoung Heo, Cheonghwa Lee et al.· International Journal of Pre...· 1 citation
Over the last decade, significant advances have been made in the field of human robot interaction (HRI), such as collaborative workcells in smart factories and autonomous mobile robots. However, for these systems to achieve widespread societal acceptance, more natural and efficient communication interfaces must be developed. This work proposes a computer vision-based human–robot cooperative system, composed of a set of gestures that, when combined into composite sequences, allow intuitive interaction with the environment. Gesture identification relies on a human pose analysis model based on deep learning, which earned the 1st place in the Flying Robots Demo at RoboCup 2025. The semantic and contextual interpretation of these sequences is conducted by Large Language Models (LLMs). Integration between visual perception and linguistic understanding enables a more expressive and adaptable form of interaction. The developed system achieved a high-fidelity gesture detection model, with average precision and recall of 0.94.
Ana Carolina Estrela, Tatiana Oliveira, Khansa Rekik et al.· International Conference on...· 0 citations
Natural and empathetic human-robot interaction is essential for social robots and other AI applications, while emotional feedback is less discussed. Thus, this paper proposes a multimodal interaction system driven by large language model (LLM). The system constructs a unified emotional state vector by integrating visual (facial expressions) and auditory (speech emotion) cues. It employs DeepSeek LLM for context-aware chain-of-thought reasoning to generate contextually appropriate verbal responses, facial expressions, and head movement commands. To achieve optimized latency and fluid embodied interaction, the system adopts a layered architecture: the upper-level LLM handles semantic understanding and behavior planning, outputting structured JSON commands; the lower-level controller generates smooth motion trajectories and manages multimodal interaction flows using a finite state machine (FSM). Experimental results demonstrate the system’s ability to effectively resolve emotional ambiguities, track emotional evolution during continuous dialogue, and achieve an optimized end-to-end response latency. This validates its feasibility and engineering value in practical human-robot interaction scenarios.
Yuxuan Chen, Chen-Yi Qiu, Ning-Chuan Wang et al.· 2026 IEEE/ASME International...· 0 citations
This work introduces a novel multi-modal orchestration framework for semantic audio-driven humanoid control, enabling robots to autonomously select and execute appropriate motion skills in real time.
J. Marcelo, M. Brienza, E. Bugli et al.· 0 citations
Human–robot interaction increasingly demands intuitive, efficient, and accessible control mechanisms, particularly for users with physical or communication disabilities. Traditional interfaces—such as joysticks, keyboards, or voice commands—often impose significant cognitive or physical effort and may be unusable for individuals with impaired speech, hearing, or motor abilities. Recent advances in artificial intelligence and computer vision offer promising alternatives by enabling robots and autonomous systems to interpret human intentions directly from visual cues. This paper introduces a vision-based control framework that allows users to operate an autonomous drone through predefined hand gestures without any physical contact with a controller. The proposed system integrates real-time computer vision with control-system engineering to translate finger poses captured by a camera into actionable navigation commands. Our method employs PoseNet for robust hand-keypoint detection, combined with a custom gesture-classification module optimized for low-latency inference. The generated gesture classes are mapped to drone control instructions, enabling tasks such as takeoff, landing, directional movement, and hovering. The development process involved coordinated work across three subsystems: (1) Data Labeling, including dataset creation and annotation using CVAT and MATLAB; (2) Robot Interface and Connectivity, focusing on reliable communication between the vision module and the drone’s flight controller; and (3) AI Model Development, comprising model selection, training, and optimization using Python, OpenCV, TensorFlow, and Google Colab. Although the project encountered initial technical and organizational challenges, the iterative development cycle ultimately led to a stable, functional prototype. Experimental results demonstrate that the system can accurately recognize gesture commands in real time and maintain responsive drone control under various lighting and background conditions. The achieved performance highlights the feasibility of replacing traditional physical controllers with AI-driven gesture interfaces, providing an accessible alternative for users who cannot operate conventional input devices. Overall, this work contributes a practical and innovative solution for enhancing human–robot interaction through contact-free control. The presented framework has potential applications not only in assistive technologies but also in fields such as rescue operations, manufacturing, and interactive robotics, where intuitive and hands-free control is advantageous. The project also offered valuable interdisciplinary experience in computer vision, robotics, and software engineering, demonstrating the effectiveness of merging AI-based perception with control-system design.
A. Majd, Mehdi Asadi, Juha Kalliovaara· AHFE International· 0 citations
Human–Robot Interaction (HRI) is a rapidly evolving research area focused on enabling intuitive, efficient, and reliable communication between humans and robotic systems. Unlike conventional robotic control interfaces, we propose a contactless control system that leverages American Sign Language (ASL) as a natural, non-intrusive, and accessible modality for remote robot operation. Our framework integrates computer vision, gesture recognition, and machine learning to achieve accurate real-time interpretation of ASL gestures and their direct translation into robotic commands. To ensure robust gesture understanding, the system employs a Spatial–Temporal Network that captures both the spatial relationships of hand and body positions as well as the temporal dynamics of gesture sequences. Recognized gestures are mapped to precise control commands that drive the motors of a robotic car, enabling responsive and accurate navigation based solely on sign-based inputs. Extensive experiments demonstrate that the proposed system achieves high gesture recognition accuracy across multiple conditions, including variations in speed, angle, and handedness, while maintaining safe and reliable robot operation. Tasks executed using this framework are performed consistently and accurately, validating its effectiveness. This paper highlights the potential of ASL-based, contactless robotic control to enhance accessibility, safety, and intuitiveness in human–robot interaction, paving the way for more natural and inclusive interfaces in autonomous systems.
Deaira Carrington, Wassila Lalouani· International Conference on...· 0 citations
