AI Literacy as Experimental Practice: Students as Investigators

Abstract

There is ongoing academic debate on whether one can teach AI literacy to undergraduate students across majors, and if yes, how. This article reports a case study: a three-week midterm project embedded in an undergraduate “AI-for-all” course. Students designed reasoning tasks, ran controlled comparisons across widely used chatbots, and evaluated both answer correctness and explanation validity. Through field experience, students with no STEM background learned what consumer chatbots can and cannot do, documenting systematic brittleness across models that “sounded right but reasoned wrong.” More critically, students built understanding of how to evaluate AI outputs. The midterm gave them agency as investigators rather than passive users. Eager to share their discoveries, they are co-authors of this article. Together, we offer here to educators and the broader scientific community a concrete example of the operationalization of AI literacy as experimental practice. The method, however, is not specific to the classroom. It shows any user how to test an AI system rather than trust it blindly. In three-week midterm project, students investigated whether AI literacy can be taught to undergrads.