Skip to content

DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

Sep 2026 · arXiv.org · Vol abs/2609.21045 · 0 citations · 59 references
Computer Science

TL;DR

Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency.

Abstract

Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/

View source

Similar papers

Preprint Sep 2026

PhyVisGen: Physically and Visually High-Fidelity Robotic Manipulation Data Generation

Large-scale manipulation demonstrations are essential for learning robust visuomotor policies, yet real-world data collection is expensive and difficult to scale. Simulation offers a promising alternative, but physical and visual discrepancies can limit the transferability of synthetic data, particularly for manipulati...

Yu Zheng, Qi-Yu Feng, Yi-Xin Wu et al. · 0 citations
Preprint Sep 2026

One Demonstration, Many Objects: Generalizing Manipulation via Local Contact Geometry

DemoMimic (Dexterous Motion Mimic), a policy that manipulates objects by focusing on their geometry local to the contact points to improve sim-to-real consistency, yielding a single real-world policy that transfers across objects of varying shape, scale, mass, and friction wherever local contact structure is preserved.

Satvik Sharma, Samrat Sahoo, Huang Huang et al. · 4 citations · ⚡1
Preprint Aug 2026

Pre-training Visual Dexterity in Simulation

Simulation Pre-training for Dexterity (SPD) is introduced, a pre-training framework for dexterous manipulation that uses data entirely collected in simulation and outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous m...

Sarthak Kamat, Adam Rashid, Satvik Sharma et al. · 0 citations
Preprint Sep 2026

Function-Preserving Data Generation for Zero-Shot Real-to-Sim-to-Real Manipulation

Robotic data generation is a promising paradigm for scaling robot learning without collecting large-scale real-world data. However, generating geometrically diverse yet physically valid data for contact-rich tasks remains challenging, especially when success depends on precise geometric interfaces. Standard shape augme...

Tian-Yi Xiang, Xupeng Xie, Jia-Hang Cao et al. · 0 citations
Preprint Sep 2026

RoboFolDeX: A Physical-World Benchmark for Long-Horizon Robotic Manipulation of Deformable Objects

RoboFolDeX is introduced, a physical-world benchmark built entirely from real-robot data, with garment folding as its primary task, and a fair real-robot evaluation platform for externally submitted policies, with standardized tasks, held-out physical objects, controlled initializations, and a unified execution protoco...

Chen-Huan Liu, Yi Xu, Feng Wu et al. · 0 citations
Preprint Oct 2026

OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes

Dexterous grasping is the foundational primitive in embodied AI, demanding massive data to train robust models. As real-world data collection is expensive, simulation has become the mainstream paradigm. Yet, while cluttered scenes best reflect real-world applications, learning to grasp within them is bottlenecked by a...

N. Fang, Zhong-Jin Luo, Yu-Xin Mo et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.