Physical AI for Cross-Embodiment Human-to-Robot Motion Retargeting with Residual Deep Koopman Learning

Abstract

Cross-embodiment human-to-robot motion retargeting requires transferring human motion to robot-feasible actions despite differences in kinematic structure, reachable workspace, and sensing-to-control latency. This paper proposes Cross-Embodiment Motion Retargeting (CE-MR), a lightweight Physical AI framework that combines two wearable inertial measurement units (IMUs), a robot-independent canonical motion representation, residual Deep Koopman (RDK) prediction, and posture-aware inverse kinematics (IK). Upper-arm and forearm orientations are used to reconstruct shoulder-relative wrist motion and arm configuration without acceleration integration or direct human-to-robot joint correspondence. Human motion is represented by a 14-dimensional state and predicted over a 200-ms receding horizon using a constant-velocity prior corrected by a compact RDK model with 2270 trainable parameters. Using 100 human-arm demonstrations across five motion categories, RDK reduced canonical-position RMSE from 0.04506 to 0.03183 at 200 ms, corresponding to a 29.36% improvement over constant-velocity prediction, while reducing full-state normalized RMSE by 28.63%. Online evaluation using the official Gazebo simulation model of the ROBOTIS OpenMANIPULATOR-X showed 100% final-target IK success for all four ablation modes. In a representative single-operator Gazebo ablation, the complete CE-MR configuration showed a mean normalized human–robot posture discrepancy of 0.0708 compared with 0.1375 in the independently performed RDK trial, while maintaining a mean Cartesian IK residual of 0.728 mm. Stable integrated execution further demonstrated operation near 20 Hz under favorable IK conditions. These results support CE-MR as a compact proof-of-concept framework for predictive human-to-robot cross-embodiment motion transfer from wearable human sensing to online simulated robot execution.