Perceived usefulness and intention to use large language model-generated feedback across three educational levels: a user-centred study in programming

Abstract

Generative artificial intelligence is increasingly used to produce formative feedback in programming education; however, whether students find such feedback useful, actionable, and worth using remains underexamined. This study presents a user-centred evaluation of feedback produced by an automated, rubric-based large language model assessment of programming responses, using a two-layer instrument, namely, response-level feedback on individual answers rated on five dimensions (clarity, specificity, accuracy, actionability, and usefulness) and a consolidated performance report rated on six dimensions (overall satisfaction, relevance, personalisation, cognitive load, intention to use, and motivation). Using a cross-sectional design, 144 students across secondary, short-cycle higher education (CTeSP—professional higher technical courses), and undergraduate programming rated 893 response-level feedback instances, and 140 of them rated 237 consolidated reports. Ratings were favourable across all dimensions, with student-level means of 4.24–4.43 (response level) and 4.11–4.38 (report level). At the response level, clarity and perceived accuracy were rated highest, and actionability and usefulness lowest. Cognitive load was the lowest at the report level. Only the between-context comparison for perceived accuracy reached statistical significance, with lower ratings in CTeSP ( ε 2  = 0.076); the small secondary sample limits conclusions about similarity. Perceived usefulness was most strongly associated with actionability and perceived accuracy ( R 2  = 0.76), and intention to use with motivation and personalisation ( R 2  = 0.53). These findings characterise the correlates of perceived usefulness and intention to use within each feedback layer, supporting human-centred designs that preserve student judgement and instructor oversight.