The Effect of Generative AI (ChatGPT) on EFL Learners’ Academic Writing Performance: A Preliminary Meta-Analysis

Abstract

Generative AI (GenAI), and ChatGPT in particular, is changing how academic writing is taught to learners of English as a foreign language (EFL). Even so, the individual studies and experiments conducted so far report results that conflict with one another, which makes the question worth examining. Aim. Our study aimed to find out the result of the generative artificial intelligence effect on writing achievement of English language learners quantifying the effect and examine the sources of heterogeneity based on the observed differences. Methods. To address the research objectives, we implemented a PRISMA 2020-compliant search protocol across Web of Science (n = 99) and ERIC (n = 200). The search incorporated targeted keywords covering topics such as AI writing tools (e.g., ChatGPT, Gemini), metacognitive and personalized learning, EFL academic writing skills, and higher/engineering education. Following a three-phase screening process and full-text assessment, nine eligible empirical studies (N = 624) with between-group post-test comparisons were selected. Standardized effect sizes (Hedges’) were extracted and pooled using a restricted maximum likelihood (REML) random-effects meta-analysis. Nine studies (N=624) were included. The effect was large and positive: g=0.92, 95% CI [0.55, 1.28], p<.001; heterogeneity was high (I²=79%). Sensitivity analyses were stable (g=0.83 with the AWE tool removed; g=0.88–0.95 across leave-one-out runs). One study (Ekmekçi & Karabulut, 2025) reported a negative effect, where teacher instruction outperformed AI. Secondary constructs (metacognition, learner autonomy) also showed a positive trend. We conclude that GenAI tends to produce a meaningful average improvement in EFL writing performance, but the outcome depends heavily on the conditions of use and on teacher guidance. Because the number of studies is small, the results should be read as preliminary.