Evaluation of Large Language Models' Capabilities in the Field of Mechanical Engineering

Abstract

This paper systematically evaluates the capabilities of fourteen Large language models(LLMs) in the domain of mechanical engineering. First, we develop a specialized test bank tailored for large models in the mechanical field, comprising various question formats including single-choice, multiple-choice, true/false, and fill-in-the-blank items. Subsequently, we conduct assessments on models including DeepSeek-v3/R1, Claude-opus-4.0, GPT-4.1, Qwen2.5, and KiMi. The evaluation results indicate that DeepSeek-v3pro and Qwen2.5 outperform other models in terms of knowledge accuracy, achieving accuracy rates of 83.72% and 79.19%, respectively.