3D Dreams Are Made of This - Comparing Sketching on XR and Tablet Devices as Input Modalities for 3D Model Generation

Abstract

Generative AI (GenAI) enables the creation of 3D objects from textual descriptions that can be used in game development or 3D printing. However, users struggle to express their vision because control mechanisms are bound to 2D space, resulting in unsatisfyingly generated objects. To address this, we introduce an Extended Reality (XR) sketching application that offers a new way to use 3D sketches to control the generation of 3D objects with GenAI. We compare this approach to a 2D sketching application on a tablet in a study (N=16), both methods using pen-based input. Results show that 3D sketching enables faster specification than 2D sketching, due to reduced mental effort and shorter ideation times. However, perceived control decreases with increasing object complexity, and this negative effect is stronger for 3D sketching. These findings highlight the potential of 3D sketching as a control modality for generative 3D object creation, while identifying object complexity as a key challenge for 3D sketching interfaces for GenAI.