MecEng, a fully automated benchmark that evaluates LLMs on the creation of multibody simulation models from parameterized textual descriptions, indicates rapidly improving, but still error-prone, mechanical engineering awareness of current LLMs.
Abstract
Large Language Models (LLMs) perform well on established code-generation and mathematical-reasoning benchmarks, but their capabilities in mechanics and spatial geometry, here denoted as mechanical engineering awareness, has not been quantified systematically. We present MecEng, a fully automated benchmark that evaluates LLMs on the creation of multibody simulation models from parameterized textual descriptions. The benchmark comprises 84 generic tasks on three difficulty levels, ranging from rigid-body systems with joints and contact to flexible multibody systems that require exact 3D geometry generation, tetrahedral finite-element meshing, and Hurty-Craig-Bampton model order reduction of machine parts. A dedicated pipeline with LLMs generates simulation-ready geometry from text using Netgen, and builds multibody system models for the code Exudyn, which are then verified against expert ground truth on several levels: system-graph isomorphism including graph node annotations, numerical solutions, and part-specific measures such as mass, geometry, and eigenfrequencies. In total, 32 open-weight and two proprietary LLMs are evaluated. On rigid-body tasks, the best open-weight model obtains an overall success rate of 86.0%, compared to 91.4% for the strongest proprietary model, while flexible multibody tasks remain considerably harder. Additional studies quantify the influence of sampling temperature, reasoning, prompt design, model size, and LLM-release date. The results indicate rapidly improving, but still error-prone, mechanical engineering awareness of current LLMs.
LLMForge is introduced, a multi-model text-to-CAD framework integrating JSON-schema validation, analytic feature scoring, mesh synthesis, and multi-round iterative refinement, studied under two critique regimes.
We present GenGX, a system that generates precise geometric diagrams from natural-language descriptions by combining large language model (LLM) interpretation with symbolic constraint solving. User prompts are translated by an LLM autoformalizer into a structured intermediate representation (IR) encoding geometric entities, relationships, and construction semantics. The IR is passed to CoreGX, a constraint solver that synthesizes a deterministic construction sequence — operating above classical Euclidean primitives — that provably realizes the specified figure without numerical optimization. The system handles classical constructions, conics, curves, and transformations, and resolves both discrete and continuous ambiguity through explicit IR specifiers and a numeric clarity optimizer that selects a visually canonical representative from any underdetermined family of valid diagrams. This hybrid architecture avoids the spatial inaccuracies endemic to purely generative text-to-image approaches, produces reproducible results, and allows users to inspect and correct the IR directly.
Kavi Wilson, P. Todd· SIGGRAPH Posters· 0 citations
Advances in large language models (LLMs) have sparked interest in automating parametric CAD modeling through natural language. Existing LLM-based approaches often treat CAD modeling as flat text generation, overlooking the hierarchical structure and geometric constraints inherent in CAD programs. We present CAD-Factory, a Text-to-CAD generation system for language-driven CAD modeling that explicitly models the structural and parametric semantics of CAD programs. Our core contribution is a new formulation of CAD generation as structured program synthesis, coupled with a learnable hierarchical CAD program representation that disentangles structural topology from parameter instantiation. Building on this formulation, CAD-Factory adopts a manager–programmer–reviewer architecture: a planning agent infers program hierarchy, a coding agent instantiates symbolic and numerical parameters, and an evaluation module enforces structural validity and geometric feasibility, which supports structure-aware reasoning, constraint-consistent generation, and interpretable program synthesis. An editor–viewer loop further enables part-aware code refinement through visual feedback, supporting iterative and controllable design workflows. We also contribute a process-oriented annotation pipeline and a Text–CAD dataset with parameter-free or partially specified descriptions that reflect real-world design expression. Extensive experiments demonstrate that CAD-Factory significantly improves structural correctness and geometric consistency across diverse CAD code generation and editing tasks, establishing a structured, interpretable, and robust framework for AI-assisted design.
Yang Liu, Daxuan Ren, Yijie Ding et al.· International Conference on...· 0 citations
Generative models have recently made significant progress in generating 3D content from image or text data, yet existing methods mostly rely on implicit or discretized geometric representations that lack parametric structure or design intent. These limitations make them poorly suited for computer-aided design (CAD). In contrast, code-based CAD representations offer parametric structure and a human-readable presentation but remain underexplored due to the scarcity of datasets. We present a dataset of over 95,000 native OpenSCAD models, primarily containing mechanical parts and engineering components, designed specifically for the programmatic generation of 3D models. The dataset consists of user-created OpenSCAD code, featuring both constructive solid geometry and sketch-and-extrude modeling, along with descriptive metadata, rendered images, and detailed license information. A five-stage pipeline ensures syntactic validity, semantic richness, de-duplication, and quality annotations. To test the dataset's potential, we fine-tuned code-generation large language models for text-to-CAD generation. The evaluation results show consistent improvements in compilation validity and geometric distance to ground-truth samples. By releasing this dataset publicly, we aim to open new opportunities for programmatic 3D model generation.
D. Fresacher, Klaus Diepold· Proceedings of the 32nd ACM...· 0 citations
Large language models (LLMs) have demonstrated a strong ability to generate syntactically correct code from natural-language specifications. In this study, we explore how LLMs can be harnessed to automatically translate a neutral graph representation of fluid system models into executable code for two widely adopted simulation environments: the Python library WNTR and the Modelica Standard Library. We conduct a systematic comparison of ten state-of-the-art LLMs and six prompting strategies that differ in the contextual information supplied (e.g., code or documentation). For each configuration we assess the generated code using a suite of software-quality metrics and we validate the functional fidelity of the resulting simulation models by reproducing benchmark fluid system scenarios. Our findings offer concrete guidance for researchers and engineers seeking to integrate LLM-driven code synthesis into model-based design pipelines. While the best-performing configurations achieve acceptable syntactic quality, we observe substantial gaps remain in simulation fidelity.
J. M. Stürmer, Jascha Knack, Tobias Koch et al.· 1 citation
Recent vision-language models (VLMs) can generate executable CAD programs from images, but existing methods mainly target coarse, general-purpose 3D objects and rarely address the fine-grained geometry and millimeter-level tolerances required in industrial mechanical design. We introduce OmniMech, the first million-scale benchmark for evaluating VLMs on executable CAD generation from industrial manufacturing data. OmniMech contains more than 251,000 fully dimensioned and toleranced 2D orthographic drawings, paired with native CAD models, multi-view renderings, mesh, STEP and B-rep representations, and rich semantic annotations. The benchmark includes four tasks: (1) parametric CAD program synthesis from engineering drawings; (2) diagram-to-3D reasoning for geometrically and structurally consistent reconstruction; (3) annotation-grounded reasoning over dimensions, symbols, feature callouts, and manufacturing constraints; and (4) tool-augmented agentic reasoning using visualization, measurement, CAD execution, and verification tools. Experiments show that current VLMs and CAD-specialized models still struggle with executable program synthesis, fine-grained 3D reconstruction, and reliable enforcement of dimensions and tolerances. We will release the benchmark data, evaluation code, and tool interfaces to support future research.
Taiting Lu, Runze Liu, Ziwei Dong et al.· 0 citations