EngiWorld provides the first rigorous foundation for measuring progress toward agents that operate professional engineering software end to end, and introduces an artifact-centric evaluation methodology built on a unified domain-verifier suite.
Abstract
Autonomous agents have made rapid progress in general-purpose computer use, yet reliable automation of professional industrial engineering remains out of reach. Engineering workflows require reasoning over geometric and physical constraints and keeping dependencies consistent across software and design stages. We present EngiWorld, a benchmark designed around the capabilities that real engineering projects require, from operating a single application to solving engineering problems end to end. It contains 1,301 expert-curated tasks spanning six engineering domains (CAD, CAE, CAM, BIM, EDA, and DCC) and 26 professional software applications, with both GUI and CLI interfaces. The tasks are organized into six types, corresponding to capabilities that real engineering projects require. We further introduce an artifact-centric evaluation methodology built on a unified suite of domain verifiers. The verifiers programmatically check whether final and intermediate artifacts are geometrically valid, physically feasible, and compliant with design rules. Beyond binary success, they also score how well optimized designs meet their specifications. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model scores only 44.7/100, and just 3.6% of runs on tasks that require cross-software coordination succeed. EngiWorld provides a rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.
It is found that weaker agents often fail before producing a valid artifact, whereas stronger agents increasingly fail on structural, geometric, and construction-process requirements, and CADWorld exposes a gap between general GUI competence and reliable execution of persistent, verifiable engineering workflows.
Zi-Han Dong, Yuan-Zhe Liu, Zhi-Yuan Ma et al.· 1 citation
SDD reconstitutes, in specification-centric form, the contracts that vibe coding dissolves: accountability, verifiability, and transferability, and outlines a research agenda for future empirical validation.
J. Díaz, J. Gayoso, Andrea Cimminio et al.· 4 citations· ⚡1
This evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes, and examines performance across domains and task information requirements, alongside the development behaviors associated with successful repairs.
P. Wang, Chen-Hao Liang, Ze-Long Xu et al.· arXiv.org· 2 citations
This work asks what a CAE simulation agent still needs beyond a generic harness, and finds that with information access and repair budget held fixed, a single-agent harness matches or beats multi-agent specialized systems.
To evaluate the frontier, we must measure models not by what they say, but by what they can engineer and build in grounded physical environments. We introduce \textsc{ArtifactArena}, an open-ended platform where models face a physically grounded hardware-software co-design challenge: engineering fully functional robots...
Kushagra Tiwary, David Mayo, Nikhil Behari et al.· 0 citations
Results show that execution alone is insufficient to characterize realistic CAD modeling and that frontier models and agents differ substantially across executability, IoUs, and visual-semantic identity.
Guan-Ling Li, Zhi-Chao Huang, Hui-Mu Yu et al.· 0 citations