From Noisy Hands to Stable Grasps: Object-Centric Dexterous Retargeting

Abstract

Retargeting human hand motions to dexterous robot hands from monocular video remains a challenging task due to the inherent noise, depth ambiguity, and frequent occlusions in vision-based hand pose estimation. Existing physicsbased methods often fail when the input reference trajectory violates physical constraints, leading to unstable optimization, severe object penetration, or loss of contact. In this paper, we propose a robust optimization framework that explicitly addresses perception uncertainty to achieve physically plausible and contact-rich dexterous manipulation. By leveraging reliable object pose estimates (e.g., from FoundationPose) as strong anchors and treating hand keypoints as soft, uncertainty-weighted targets, our method recovers feasible robot trajectories even from noisy visual inputs. We introduce a contact-implicit refinement strategy that utilizes the geometric relationship between the hand and the object to dynamically correct hand pose errors. Furthermore, we integrate a physics-consistent post-processing stage to ensure temporal smoothness. We evaluate our approach on a self-collected monocular hand-object interaction dataset and the public OakInk benchmark using the Inspire Hand. Experimental results show clear gains over both kinematic IK and SPIDER, with substantially reduced penetration depth, improved contact stability, and smoother joint trajectories, bridging the gap between noisy visual perception and precise robotic control.