Beyond Guessing: Task-Level Predictors of Non-Native English-Speaking Students’ Ability to Identify AI-Generated Scientific Texts
TL;DR
How recognition accuracy varies across texts is analyzed and which characteristics make AI-generated physics texts more or less distinguishable from human-written ones are discussed, with implications for fostering AI literacy in physics education.
Abstract
The increasing prevalence of artificial intelligence (AI), in particular large language models (LLMs), is transforming how scientific texts are produced and consumed. A growing share of online content is at least partially generated by AI, often without readers’ awareness, raising the question of whether students can recognise such texts and under what conditions. In this study, 372 high school students evaluated short physics texts and indicated whether each had been written by a human author or by ChatGPT. The texts were organised into tasks that varied in topic and other task characteristics, including cognitive demand. Overall, students struggled to identify AI-generated texts, performing only slightly above chance. Focusing on task-level features, we analyse how recognition accuracy varies across texts and discuss which characteristics make AI-generated physics texts more or less distinguishable from human-written ones, with implications for fostering AI literacy in physics education.