The MineCEraft benchmark comprises 723 domain-expert hand-crafted natural-language instructions with programmatically verifiable evaluation, providing a safe and controllable experimental environment for assessing LLMs'ability to perform realistic construction engineering tasks.
Abstract
We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source benchmark designed to systematically evaluate the reliability and limitations of LLMs for construction tasks in Minecraft. The MineCEraft benchmark comprises 723 domain-expert hand-crafted natural-language instructions with programmatically verifiable evaluation, spanning 17 distinct task categories, providing a safe and controllable experimental environment for assessing LLMs'ability to perform realistic construction engineering tasks. With this benchmark, we conduct an in-depth evaluation of state-of-the-art LLMs and perform a detailed error analysis, revealing key failure modes and practical challenges in applying LLMs to construction engineering tasks.
This work proposes StrategyBench, which selects strategy-inducible tasks from BIG-Bench, constructs reference strategies, and defines evaluation metrics along two dimensions: strategy quality and downstream utility, and experiments show that explicit strategy utility differs substantially across task categories and dep...
Jing-Han Tan, Yuanzhe Wang, Lu Chen et al.· 0 citations
This paper systematically evaluates the capabilities of fourteen Large language models(LLMs) in the domain of mechanical engineering. First, we develop a specialized test bank tailored for large models in the mechanical field, comprising various question formats including single-choice, multiple-choice, true/false, and...
Wen-Rui Zhang, Zhao-Xi Fang· 2026 3rd International Confe...· 0 citations
Large language models are paving the way towards innovation by understanding, analyzing, summarizing and generating content in the modern world. Currently there are thousands of LLM projects developed by engineers in open-source repositories. However, whether these LLM projects have underlying patterns or not remains a...
AI large language models (LLMs) like ChatGPT can improve the efficiency of common language generation tasks performed by engineering professionals. In this session, we draw upon emerging academic research to suggest applications of LLMs in the field of industrial and systems engineering. Our demonstrations will showcas...
Joseph H. Wilck· IISE Annual Conference &...· 0 citations
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluati...
The study examines Large Language Model-generated answers to 86 text-based questions covering three task categories chosen for their direct correspondence to typical engineering design activities to characterize the specific capabilities that AI-enabled design systems must develop to be reliable in practice and establi...
M. Mysior, C. Iniotakis· Research in Engineering Desi...· 1 citation
