College students' metacognitive awareness of generative-AI reliance: an experimental study of decision confidence and attribution bias

Abstract

Introduction Generative AI is increasingly used by college students for explanation, evaluation, and decision support, raising the question of whether perceived reliance accurately tracks the extent to which AI advice shapes final judgments. Methods In a three-condition between-subjects experiment, 342 undergraduate students completed reasoning and information-evaluation tasks under independent decision-making, open multi-turn ChatGPT support, or the same ChatGPT support with a brief metacognitive reflection prompt. Participants reported initial and final answers and confidence, and AI-assisted chat logs were coded for the final recommendation, its correctness, and behavioral reliance. Results Open ChatGPT support increased final confidence and was associated with 62.4% acceptance of incorrect AI advice. Reflection reduced incorrect-advice acceptance to 39.7% (OR = 0.40, 95% CI [0.28, 0.56], p < 0.001), improved awareness calibration (0.59 vs. 0.41, p < 0.001), and reduced the attribution-bias index (0.21 vs. 0.42, p = 0.002). Recommendation accuracy was comparable between the two AI-assisted groups. Discussion Brief metacognitive reflection improved immediate discrimination between useful and misleading AI advice without producing blanket rejection of AI. The exploratory association between calibration and overreliance is not interpreted as causal mediation, and the findings concern bounded academic decision tasks rather than long-term learning.