Assessing the Academic Performance of ChatGPT in Undergraduate Civil Engineering: A Study on Material Science and Building Materials Exam

Abstract

With the emergence of Large Language Models (LLMs), significant advancements have occurred in artificial intelligence technologies. Models such as ChatGPT, developed by OpenAI, stand out for their high performance in areas like accessing information, writing code, assisting in data analysis, translation and multilingual communication, solving complex problems, and creating content; in these respects, they offer innovative applications across many fields. The primary objective of this study is to comprehensively evaluate the performance of ChatGPT, one of the leading LLMs, in answering undergraduate-level exam questions in the field of civil engineering, which requires expertise and technical knowledge. For this research, the courses of "Material Science" and "Building Materials," sub-disciplines of civil engineering, were selected, and exam questions of varying difficulty levels and types were used. The responses provided by ChatGPT to these questions were evaluated by the course lecturers based on pre-determined scoring criteria. The scores of ChatGPT and the students from the same exams were analyzed using statistical methods. To compare their performances, z-scores, percentiles, and t-tests were utilized for all exam scores. In the evaluations for the Material Science course, ChatGPT demonstrated high accuracy and technical proficiency in its responses to questions involving definitions, conceptual explanations, and multiple-choice formats. According to the t-test results, ChatGPT's average scores for the midterm, final, and raw score were significantly higher than those of the students. For the Building Materials course, ChatGPT's answers to multiple-choice and conceptual open-ended questions were generally found to be accurate, explanatory, and coherent. However, significant limitations were observed in questions requiring graph generation and visual interpretation. The t-test results also showed that ChatGPT's average scores for the midterm, final exam, and raw score were significantly higher than the students' for this course. This study reveals the academic potential and limitations of ChatGPT within the field of civil engineering. Furthermore, it aims to shed light on future studies and discussions regarding the place of ChatGPT in civil engineering education.