ESTHER is proposed, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo, and shows state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods.
Abstract
Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.
EgoFound3R, a unified end-to-end model that estimates world-space hand geometry in a metric scale shared with the scene, and predicts point-wise interaction attributes, including visibility, contact, and distance, is introduced.
Hong-Ming Fu, Jing-Cheng Shi, Wen-Jia Wang et al.· 0 citations
MEgoVista is offered as a measured route from egocentric video to metric hand supervision, one that widens where such labels can be gathered, one that widens where such labels can be gathered.
Jian-Gong Xiao, Zhi-Hao Zhang, Yi-Fei Dong et al.· 0 citations
Invariant human perception from visual input is essential during human-robot interaction. Although multiple approaches exist, such as face recognition or human pose estimation, these methods frequently fail in real-world interactive settings, where robots typically operate from close-range view-points that result in pa...
Anna L. Lange, Bo-Song Ding, M. Kirtay et al.· IEEE International Symposium...· 0 citations
Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation.
Kai Guan, Minchao Jiang, Ruichen WangLi et al.· 0 citations
Egocentric human video offers a scalable data source for dexterous manipulation, yet using it to train humanoid robots presents two challenges: (1) an embodiment gap, as human hands differ structurally from robot end-effectors and low-cost egocentric recordings lack the torso kinematics required by conventional retarge...
Hui-Min Pan, Y. Ren, Kun-Peng Song et al.· arXiv.org· 0 citations
EgoWild2Dex, which transfers in-the-wild ego-human experience to dual-arm robots with dexterous hands by jointly aligning unstable egocentric views and human motions with robot observations and actions, is introduced and GeoFormer, a differentiable geometric transformer that warps noisy human observations toward robot...
Kun-Li Channing Lin, Xu-Tao Wen, Jing Lin et al.· arXiv.org· 0 citations
Known for his clear and elegant writing style, Bertsekas shaped fields from control and optimization to large-scale computation and artificial intelligence.