Beyond LLM output: how critical thinking shapes EFL students’ interaction with ChatGPT generated content

Abstract

Abstract The increasing incorporation of Artificial Intelligence (AI) tools in education urges researchers to examine their efficiency in Foreign Language (FL) learning and teaching. Large Language Models (LLMs) such as ChatGPT are becoming inevitable in promoting English as a Foreign Language (EFL). However, concerns have been raised about overreliance on LLMs, which may hinder learners’ Critical Thinking (CT) abilities. This study explores the use of ChatGPT to develop students’ capacity for critically engaging with LLM-generated content and assessing their abilities in reflective learning, fact-checking, logical reasoning, and bias detection. Using convergent parallel mixed methods, including quantitative rubric-based assessment and qualitative response analysis, students’ interactions with the LLM are evaluated based on Paul and Elder’s (2013) CT nine intellectual standards: clarity, accuracy, precision, relevance, significance, depth, breadth, logic, and fairness. Findings reveal that while advanced students critically reflect on AI responses, lower-proficient learners often adopt them uncritically, highlighting a gap in fact-checking and AI literacy. Many students also struggle to detect inconsistencies or bias, underscoring the need for explicit instruction on AI ethics and limitations. The study concludes that AI is useful for language learning, but its effectiveness depends on learners’ ability to critically evaluate and refine its outputs.