Skip to content

AI Literacy as Experimental Practice: Students as Investigators

Aug 2026 · Communications of the ACM · Vol 69, pp. 48-56 · 0 citations · 3 references

TL;DR

A three-week midterm project embedded in an undergraduate “AI-for-all” course investigated whether AI literacy can be taught to undergrads, and shows any user how to test an AI system rather than trust it blindly.

Abstract

There is ongoing academic debate on whether one can teach AI literacy to undergraduate students across majors, and if yes, how. This article reports a case study: a three-week midterm project embedded in an undergraduate “AI-for-all” course. Students designed reasoning tasks, ran controlled comparisons across widely used chatbots, and evaluated both answer correctness and explanation validity. Through field experience, students with no STEM background learned what consumer chatbots can and cannot do, documenting systematic brittleness across models that “sounded right but reasoned wrong.” More critically, students built understanding of how to evaluate AI outputs. The midterm gave them agency as investigators rather than passive users. Eager to share their discoveries, they are co-authors of this article. Together, we offer here to educators and the broader scientific community a concrete example of the operationalization of AI literacy as experimental practice. The method, however, is not specific to the classroom. It shows any user how to test an AI system rather than trust it blindly. In three-week midterm project, students investigated whether AI literacy can be taught to undergrads.

View source

Similar papers

Open access 2026

Education isn't fun anymore, is this why students use AI?

In contemporary academic environments, students are increasingly employing artificial intelligence (AI) for purposes that extend beyond traditional support functions such as shaping ideas and correcting grammar or spelling. While much of the existing discourse focuses on concerns around increased efficiency, reduced student engagement, and the erosion of authentic learning, this research reframes the issue from a broader perspective: the enjoyment of the educational experience. Rather than positioning AI solely as a threat to engagement, this study explores what students themselves find meaningful and engaging in their learning. Insights are drawn from end-of-year student feedback alongside discussions with academics participating in the “Universities Pull the Plug” Signal group. These data suggest that students respond positively to learning environments that are interactive, socially engaging, and intrinsically motivating. Within this context, the research investigates the role of game-based learning as a mechanism for enhancing engagement and enjoyment. Specifically, it examines the application of the PlanIT Sustainable Development Game as a pedagogical tool that integrates entertainment with the practical application of knowledge and skills acquired during teaching. Student feedback following gameplay is used to evaluate how such approaches influence engagement, understanding, and perceived relevance of course content. The overarching aim of this study is to develop a narrative and strategic framework for student engagement that emphasises enjoyment, active participation, and personal development. By fostering environments where students experience intrinsic motivation and recognise the value of their own skills development, the research seeks to encourage deeper engagement with learning processes—independent of reliance on AI tools.

John F. Grant · 0 citations
Open access Jul 2026

A Metacognitive Blind Spot: Student Comprehension, AI Reliance, and the Conceptual Difficulty Gap

This exploratory study investigates the relationship between student metacognition, use of artificial intelligence, and empirical performance within a multidisciplinary course on AI. Using data from a sample of college-age, full-time undergraduate students (averaging 18 participants per assessment) enrolled in an in-person junior seminar at a Midwestern U.S. university, we correlate student self-assessments with standard readability metrics (e.g., Flesch–Kincaid), L2SCA metrics, and objective assessment outcomes, analyzing how learners evaluate their own comprehension and how they deploy AI tools in response to the complexity of 21 reading assignments over 8 weeks. We find that students’ perceptions of linguistic difficulty correlate with classical readability scores, but their perceptions do not predict their success nor does their engagement with assistive AI. The results suggest that students utilize generative AI tools as a habitual baseline rather than a strategic response to difficult material. We argue that while students can identify surface-level linguistic friction, they fail to recognize deep conceptual hurdles, leading to a false sense of mastery that neither their intuition nor their AI assistants appear to mitigate. We propose that a quantifiable metric, the Conceptual Difficulty Gap (CDG), may be useful for identifying a class of texts that syntactically appear to be simple, but consistently trigger performance failures. Crucially, we uncover a possible metacognitive blind spot: student self-ratings of difficulty are negatively correlated with this gap, implying that student assessments of difficulty are not based on actual conceptual difficulty. Furthermore, self-reported AI reliance shows no correlation with the gap, indicating that students may not be strategically deploying generative AI tools to mitigate conceptual difficulty.

Igor Crk, E. Gultepe · 0 citations
Preprint Jul 2026

Experimental Evidence on the Learning Impact of Generative AI

Evidence is found for two mechanisms behind the learning gains: students shift time away from drafting text and toward reading and searching for information, and they report greater learning enjoyment.

Zara Contractor, Germán Reyes · 1 citation
Open access Jul 2026

SCAFFOLD OR SHORTCUT? THE CHOICE OF AI TOOLS AS A PEDAGOGICAL ACT

This article examines the relationship between undergraduate students’ choice of generative artificial intelligence (AI) tools and the cognitive operations mobilized in scientific research, drawing on Vygotsky’s cultural-historical theory and the concept of the Zone of Proximal Development. It is based on an exploratory diagnostic study, with a mixed-methods approach, conducted with 166 undergraduate students at the Federal University of Mato Grosso do Sul, through a questionnaire with closed questions and open fields, treated by descriptive statistics and qualitative reading. The results indicate intensive use of AI — 72.3% of respondents use these tools weekly or daily — yet concentrated in general-purpose generative systems: among those who use AI in any research phase, 95.6% rely exclusively on generative or language-revision systems, with residual presence (2.4%) of AI-layered scientific databases. The open questions reveal concerns about reliability, plagiarism, impoverishment of one’s own learning, and the lack of institutional guidelines. The mismatch between intensity of use and functional adequacy of the tool is discussed as evidence of an unmediated zone of development, in which students build inadequate scaffolds on their own. The article concludes that AI should be understood as a mediating instrument, not as a more capable peer — a role that remains reserved for the teacher, who must mediate the very choice of the tool. An analytical mediation matrix is proposed to guide this teaching practice.

Heloísa Portugal, Carolina Ellwanger · 0 citations
Jul 2026

Student-facing conversational AI in primary classrooms: Should it be allowed?

This paper examines the opportunities and risks associated with student-facing conversational artificial intelligence (AI) in primary education. It aims to evaluate how large language models (LLMs) can support personalised learning while identifying developmental, pedagogical and ethical challenges. Rather than treating benefits and risks as discrete factors, the study conceptualises AI as a socio-technical intervention that reshapes relationships between learners, teachers and knowledge. The paper adopts a conceptual and theory-driven approach, synthesising current literature on AI in education, pedagogical theories and emerging practices in primary classrooms. The analysis is structured through a tension-oriented synthesis, identifying points of alignment and misalignment between AI affordances and core learning processes in primary classrooms. Based on this synthesis, the study develops a set of guiding principles grounded in developmental and educational considerations. Conversational AI offers significant benefits, including personalised learning support, immediate feedback and reduced teacher workload. However, risks include cognitive offloading, overreliance on AI, misalignment with curriculum goals and ethical concerns such as bias and privacy. The analysis suggests that these are not independent challenges but reflect underlying tensions between technological capabilities and pedagogical requirements. The study is conceptual and lacks empirical validation. Future research should focus on longitudinal and classroom-based studies to assess the actual impact of AI on primary learners' cognitive and social development. The paper highlights the need for interdisciplinary research bridging education, AI and developmental psychology. The study proposes a set of guiding principles derived from the identified tensions, emphasising teacher-mediated interaction, developmental calibration of AI use, transparency, curriculum alignment, privacy protection and equity considerations. These principles provide a structured basis for integrating AI in ways that support learning processes while mitigating potential risks. The adoption of AI in primary education raises concerns about equity, access and digital divides. Without careful implementation, AI may reinforce existing inequalities. Promoting critical AI literacy and ethical awareness among young learners is essential to prepare them for responsible participation in an AI-driven society. This paper contributes a developmentally informed, tension-based conceptual framework for understanding student-facing AI in primary education. By reframing commonly identified opportunities and risks as interrelated tensions, it offers a more analytically grounded basis for guiding AI integration beyond descriptive or normative approaches.

Misbah Zulfiqar, Umair Iqbal · 0 citations
2025

The Read-Reflect-Respond (R3) Framework: Designing AI-resilient Assignments to Promote Authentic Learning and Academic Integrity

The emergence of AI in education has created both opportunities and challenges, especially in students’ examinations and assessments. Today, AI can solve almost any problem in seconds and provide answers in any style. This is useful in education, but its lead to misuse of AI in assignments and examinations, where students can solve the questions through AI, without independently thinking about the questions. AI provides simpler solutions, according to the prompt and can mimic human-like responses. Using AI to solve assignment questions has posed a challenge to the development of creative and critical thinking. Recently, students are directly copying AI-generated texts and pasting or writing in their answer sheets. Although AI has the potential to solve any problem, it has become a challenge for educators to evaluate ethically. There are several tools available, like plagiarism detection tools and AI-content detectors. This paper proposes the idea of AI-resilient assignments maintaining academic integrity with product- and process-based evaluation approach. These AI-resilient assignments are similar to the open-book system, where they require human intelligence, independent thinking, and personal understanding to solve them correctly.

Atul Sahu, Chandrakant Kumar Singh, A. K. Malik · 0 citations