It is found that weaker agents often fail before producing a valid artifact, whereas stronger agents increasingly fail on structural, geometric, and construction-process requirements, and CADWorld exposes a gap between general GUI competence and reliable execution of persistent, verifiable engineering workflows.
Abstract
Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engineering workflows whose outputs are persistent, structured artifacts. Mechanical computer-aided design (CAD) is a particularly demanding setting: an agent must manipulate geometry and constraints over long interaction horizons while producing a native project whose dimensions, construction structure, and downstream engineering state remain valid. We introduce \textbf{CADWorld}, a benchmark for long-horizon computer use in FreeCAD. CADWorld contains 200 tasks spanning 11 mechanical-CAD workflow categories, including sketching, part modeling, assembly, CAM, FEM, measurement, mesh processing, and technical drawing. Agents operate through screenshots and GUI actions, while success is determined by task-specific executable checks over saved FreeCAD artifacts and auxiliary outputs, covering geometric properties, parametric structure, constraints, manufacturing state, and simulation results. Across seven current agents on the full benchmark, the strongest agent achieves 17.5\% success, compared with an 87.0\% expert reference pass. We find that weaker agents often fail before producing a valid artifact, whereas stronger agents increasingly fail on structural, geometric, and construction-process requirements. CADWorld therefore exposes a gap between general GUI competence and reliable execution of persistent, verifiable engineering workflows. Project accessible at https://cad-world.github.io.
Results show that execution alone is insufficient to characterize realistic CAD modeling and that frontier models and agents differ substantially across executability, IoUs, and visual-semantic identity.
Guan-Ling Li, Zhi-Chao Huang, Hui-Mu Yu et al.· 0 citations
EngiWorld provides the first rigorous foundation for measuring progress toward agents that operate professional engineering software end to end, and introduces an artifact-centric evaluation methodology built on a unified domain-verifier suite.
Hong-Cheng Gao, Hai-Long Qu, Yu-Ang Lei et al.· 0 citations
Editable CAD generation requires access to procedural history rather than final geometry alone, yet widely used CAD datasets typically omit explicit geometric constraints, native parametric assets, or rich semantic labels. We introduce EPICCAD, a multimodal benchmark for history-based CAD modeling that aligns compact c...
Recovering executable CAD programs from 3D meshes is challenging due to the compositional nature of CAD construction and the interaction between discrete modeling choices and continuous parameters. Many learning-based methods predict complete programs in a single pass and rely predominantly on sketch-extrude representa...
Advanced structural and materials design increasingly depends on complex geometries and microstructures spanning continuum and atomistic descriptions. Yet the early idea–design–simulation loop remains manual: engineers translate informal intent into modeling assumptions, select physics, assemble solver inputs, and revi...
Jie Tian, Lin Pang, Ji-Xin Hou et al.· Advanced Intelligent Systems· 0 citations
This work proposes CIT-CAD, a framework that infers a Constraint Intent Tree (CIT) from the input description to represent the intended entities, hierarchy, operations, and relations in CAD code generation and defines expected constraints for verification.