Generative Artificial Intelligence, Perceived Effort, and Critical Thinking: Evidence from an Experimental Study with ChatGPT in Higher Education

Abstract

The emergence of Artificial Intelligence (AI) has had a profound impact on diverse sectors, including education. While the integration of technology into classrooms has been a constant for decades, both educators and students face new challenges due to the advent of NLP systems. Educators deal with analyzing and evaluating the originality of an essay. With the application of AI, students can answer theoretical questions and solve complex exercises, but grapple with the challenge of collaborating and engaging with a tool that relies on natural language instructions but can also provide outright incorrect responses. Our study aims to assess the effectiveness of ChatGPT in solving a case study with quantitative questions. We seek to compare the performance of students who use the tool with those who do not, from formulating instructions to validating or assessing the results. The data collected through pre-, and post-test questionnaires reveal consistently better results in students who did not use ChatGPT. However, the expectation of result accuracy and confidence in the outcomes obtained are similar in both groups. We suggest this to be interpreted as an effect that we call "Blind to Fraud" describing the situation in which students appear unaware of the likelihood of inaccuracy when using an AI-based NLP tool, which shows overconfidence in the results obtained. This can significantly impact learning outcomes and should be counterbalanced through critical thinking skills in education.