Skip to content
Preprint

MIRA: Real-Time Full-Duplex Human-Robot Interaction for Embodied Companions

Sep 2026 · 0 citations · 43 references
Computer Science

TL;DR

This work presents MIRA, a unified framework for real-time full-duplex embodied companion interaction, and designs CORTEX, a dual-timescale interaction policy that combines low-latency barge-in preemption and streaming response generation with deliberative turn decisions, backed by a robot-side execution layer that enforces physical safety constraints during execution.

Abstract

% !TEX root = ../main.tex Real-time embodied companion interaction requires a robot to infer user intent from streaming speech, generate timely responses, and execute expressive, interruptible motions. Existing systems typically decouple dialogue orchestration from gesture synthesis, relying on offline motion generation from complete audio. This separation leaves open how a deployed robot can dynamically synchronize response content, prosodic timing, and physical safety under incremental inputs and uncertain turn boundaries. We present MIRA, a unified framework for real-time full-duplex embodied companion interaction. Given streaming user speech, dialogue history, and vocal affect, MIRA predicts both the response text and an explicit embodiment cue that routes the response to the appropriate physical behavior. Discrete social behaviors (\eg listening and greeting) are mapped to validated robot trajectories, while speaking responses are accompanied by streaming, generative co-speech motion. For co-speech motion generation, we propose ROSCO, a prefix-conditioned diffusion model for streaming audio-to-joint motion generation. We further design RHPC, an inference scheme that maintains a sufficiently long temporal context for motion prediction while bounding physical commitment to a short, interruptible prefix. At the interaction level, we design CORTEX, a dual-timescale interaction policy that combines low-latency barge-in preemption and streaming response generation with deliberative turn decisions, backed by a robot-side execution layer that enforces physical safety constraints during execution. MIRA is deployed on an Astribot S1 humanoid robot. Quantitative evaluations demonstrate competitive audio-motion alignment relative to state-of-the-art motion-generation baselines, while real-robot deployment measurements characterize streaming responsiveness and interruption handling.

View source

Similar papers

Preprint Sep 2026

SocialHumanoid: Towards Expressive Humanoid Behavior via One-Step Co-Speech Motion Generation

Perceptual evaluations further show that training with AffectMoCap improves affect recognition from generated body motion, while real-robot experiments demonstrate continuous affect-conditioned behavior and stable long-horizon execution.

Cheng-Qun Yang, Teng-Jie Zhu, Liang Xu et al. · 0 citations
Preprint Oct 2026

Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot

Expressive humanoid interaction requires speech, facial animation, and body gestures to form a coherent response. However, many full-body humanoid robots produce speech and gestures without a visually expressive face, while talking-face animation and robot gesture generation are typically developed separately. We prese...

Jin Jiang, Kun Li, Jian Ma et al. · 0 citations
Review Sep 2026

EmoPose: Vision-Language Model Guided Emotion-Aware Gesture Generation for Humanoid Robots

Socially competent humanoid robots must communicate affect and intent through gesture as well as speech, yet open-ended interaction must become motion that is both expressive and executable on a specific body. This demands semantic flexibility for contextual social intent while preserving deterministic, embodiment-awar...

Dao-Jie Peng, Bing-Tao Wang, Fu-Long Ma et al. · 0 citations
Book Open access Aug 2026

Intent-Aware Backchanneling for Active Listening in Human-Robot Interaction

In conversational Human-Robot Interaction, robots typically remain silent during user speech and reply only after a pause, making interaction feel unnatural. In contrast, humans signal that they listen through active behavior. To overcome this, we present a system in which a social robot conveys active listening throug...

Yang Sun, Jan Leusmann, Michael A. Hedderich · 0 citations
Conference Aug 2026

Beyond the Prompt: Why Physical Embodiment Still Matters for LLM-Powered Conversational Agents

Large Language Models (LLMs) have transformed conversational AI, but an unresolved human-robot interaction question remains: “With conversational agents, does physical embodiment matter?”. This paper compares an embodied Pepper robot with a non-embodied laptop interface driven by the same LLM - Llama 3.2 7B parameter m...

Mriganka Biswas, John Murray · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.