CAD2Real: CAD-Grounded Skill Primitives and Vision-Language Planning for Precision Assembly

Abstract

Putting together a multi-part assembly feels routine in human hands: we visually align parts, feel first contact, correct tiny misalignments, and decide what to do next from the evolving scene. For a robot, precision assembly is a long-horizon, contact-rich problem where millimeter-level pose errors or a mistimed action can lead to jamming, binding, or irreversible failure. While CAD assets offer clean geometry for training, deployable performance is limited by a persistent sim-to-real gap in both perception and execution. We present CAD2Real, a hierarchical robot learning framework for precision assembly that converts a modest number of human demonstrations and teleoperated rollouts into scalable, deployable skills. CAD2Real (i) decomposes long-horizon assembly into parameterized action primitives, (ii) trains primitive policies with CAD-derived simulation geometry paired with teleoperation data, and (iii) at deployment uses a vision-language foundation model to sequence and parameterize primitives from the current geometric state. Across diverse multi-part assemblies and variations in part geometry, CAD2Real transfers robustly to the real world, generalizes from limited demonstrations, and supports low-shot adaptation to novel object configurations.