EgoWild2Dex, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, is introduced and GeoFormer, a differentiable geometric transformer that warps noisy human observations toward robot observations, is introduced.
Abstract
Egocentric human data provide a principled source of supervision for learning dexterous robot manipulation. Unlike prior approaches that often collect such data in constrained or specially constructed environments, we collect in-the-wild egocentric demonstrations in real-world settings, including homes, factories, and pharmacies, etc., where people perform their ordinary tasks while wearing head-mounted cameras. This collection protocol captures diverse workflows and hand-object interactions across long-tailed object and skill distributions, but also yields visually challenging observations due to scene clutter and head-motion-induced viewpoint changes (a mean cumulative rotation of $15.93^{\circ}$/s). To address these issues, we introduce EgoWild2Dex, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, respectively. This work offers three benefits. First, we introduce GeoFormer, a differentiable geometric transformer that warps noisy human observations toward robot observations. Second, we design a human-robot training scheme to bridge the embodiment gap, enabling high task success with limited robot supervision. Third, we release EgoWild, a 538.9-hour in-the-wild egocentric human dataset comprising 179,049 episodes, 125,961 unique task descriptions, and 1,282 object categories. On real robots, EgoWild2Dex achieves an average success rate of 96.7% across three long-horizon bimanual dexterous manipulation tasks and an average object-level zero-shot success rate of 33.3%. The data, models, and code will be released.
HandEdit serves as a critical resource at the intersection of image editing and robotics: it advances embodiment-aware editing models while enabling scalable dexterous robotic learning from abundant human video data, paving the way for more generalizable Embodied AI.
Zhen-Jie Yang, Xingyu Jiao, Guopeng Zhong et al.· 5 citations
Experiments show that joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types, with benefits validated on real-robot deployment.
Ye Wang, Peibin Lin, Xiong-Hui Chen et al.· 13 citations· ⚡1
This work presents RoboReact, a framework that automatically synthesizes whole-body humanoid manipulation skills from a single egocentric RGB-D observation, and highlights the potential of combining generative models, vision-language reasoning, and closed-loop control for scalable humanoid skill acquisition.
Shu-Liang He, Shuai Wang, Bo Yue et al.· 3 citations· ⚡1
Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning.
Mobile bimanual dexterous manipulation requires continuous coordination of locomotion, whole-body motion, and finger-level dexterity within a single trajectory, creating a severe robot demonstration bottleneck. Egocentric human demonstrations offer a scalable alternative, but prior approaches ease the transfer by simpl...
Rui Zhou, Yi-Bo Yuan, Jun-Kai Zhao et al.· 0 citations
This work presents RobotMover, a complete learning-based system for large-object manipulation that leverages human–object interaction demonstrations to train robot control policies and achieves strong performance in terms of capability, robustness, and controllability, outperforming both learned and teleoperation basel...
Tian-Yu Li, Joanne Truong, Tsung-Yen Yang et al.· IEEE Transactions on robotic...· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.