Aug 2026· Proceedings of the 2026 ACM Conference on International Computing Education Research Vol. 1· pp. 59-72· 0 citations· 55 references
TL;DR
This paper presents the development and initial validation of an instrument to measure self-efficacy while using GenAI to learn programming, and finds strong support for the validity of the existing Steinhorst instrument in a new context, specifically an introductory programming course that fully integrates GenAI.
Abstract
The rise of generative artificial intelligence (GenAI) has sparked a rapid change in computing curricula and teaching approaches. GenAI coding tools can accurately complete assignments, answer test questions, and perform other tasks traditionally associated with learning programming, especially at the introductory level. Because GenAI is still so new, researchers investigating student usage of GenAI have used informal rubrics and questionnaires. To advance, the field needs validated instruments that measure student perception and use of GenAI. This paper presents the development and initial validation of an instrument to measure self-efficacy while using GenAI to learn programming. Self-efficacy is an important construct in education research because it robustly correlates with student success, across disciplines and ages, including undergraduate computing education. Computing education researchers have presented several validated self-efficacy instruments, most recently by Steinhorst et al. in 2020. Critically, this instrument was created before the rise of GenAI’s popularity in 2022. To complement this instrument, we created a GenAI scale similar in style to the Steinhorst self-efficacy instrument, consisting originally of 11 items and revised to 5 items. We report two important findings in this paper. First, we found strong support for the validity of the existing Steinhorst instrument in a new context, specifically an introductory programming course that fully integrates GenAI. Second, the new GenAI scale shows strong internal reliability, discriminant validity with items in the Steinhorst subscales, and criterion validity with students’ GenAI usage patterns. Based on statistical analysis and cognitive probing interviews, we argue for the validity of the five-item scale to measure students’ GenAI self-efficacy in the context of programming.
The rapid integration of Generative Artificial Intelligence (GenAI) into programming education has raised important questions regarding its impact on learning processes, conceptual understanding, and technological dependency. This study analyzed the effects of four GenAI-supported instructional strategies in an introductory programming course for undergraduate engineering students. A multi-group quasi-experimental pre-test–post-test design was implemented involving 686 students distributed across 53 class groups, from 10 campuses, taught by 32 professors. The instructional conditions included Quizzes for Self-Regulation, Github-Copilot-assisted learning, Prompt Problems with Iterative Refinement, and Flipped Learning enhanced with GenAI, which were compared against a traditional teaching approach. Learning outcomes were measured using normalized learning gain, while statistical analyses were conducted using non-parametric methods due to deviations from normality and heteroscedasticity. Results indicate that GenAI integration did not produce statistically significant overall differences in learning gain when all GenAI-supported strategies were analyzed as a single cluster compared to traditional instruction. However, differences emerged between specific strategies, with Quizzes and Copilot-based approaches having higher median learning gains than Prompt Problems and Flipped Learning strategies. No statistically significant differences associated with gender were identified. These findings suggest that the effectiveness of GenAI in programming education depends less on the mere presence of the technology and more on the pedagogical conditions under which it is integrated into the teaching–learning process.
G. Huesca, Y. Martínez-Treviño, Claudia Gabriela Jiménez González et al.· Applied Informatics· 0 citations
This study used a convergent parallel mixed-methods design to examine at how generative Artificial Intelligence (AI) is reshaping educational practices, however there is limited evidence regarding its impact on students’ learning outcomes, especially in developing countries. This study further examined the relationships among AI-related skills, perceived usefulness, self-efficacy, motivation, and academic performance, as well as students’ experiences with generative AI in learning environments. Applying a survey design, this research collected survey data from two hundred ninety-nine ( N = 299) students in Nepal, Indonesia, and Brazil. This study also conducted semi-structured interviews with fourteen ( N = 14) interviewees. Quantitative results indicated low levels of AI literacy, self-efficacy, perceived usefulness, and motivation, all averaging means below midpoint scores of five point Likert scale of survey variables. Regression analyses revealed weak correlations between those independent variables and reported academic enhancement, suggesting that students often lack the confidence and skills required to integrate AI tools effectively into their learning activities.
Conversely, qualitative results highlighted significant advantages of generative AI, such as improved learning efficiency, enhanced communication, and greater problem-solving capabilities. Interviewees noted that AI tools simplified complex concepts and saved time, despite receiving limited formal training. The integration of quantitative and qualitative results exposed a perception practice gap: while students benefited from generative AI, they generally underestimated their abilities and lacked structured guidance. The study concludes that successful AI integration in education required more than mere access to technology; it requires fostering AI literacy, self-efficacy, motivation, and ethical awareness through dedicated pedagogical support.
Basanta Prasad Adhikari, Suyantiningsih, Ariyawan Agung Nugroho et al.· OCEM Journal of Management,...· 0 citations
In this research-to-practice paper we present a survey that can be used to assess students'AI knowledge. As the use of artificial intelligence (AI), including generative artificial intelligence (GenAI), has proliferated, so has the need to educate students about the topic. A range of AI literacy frameworks have been proposed, outlining the essential knowledge that students should have. Alongside, different ways of assessing AI knowledge have been developed. As yet, there is a lack of assessment instruments capable of evaluating multiple forms of student knowledge, including technical concepts, practical applications, and ethical concerns about AI use. In this article, we present a study implementing a comprehensive instrument to assess AI knowledge. The instrument combines measures from multiple scales to capture a range of literacy features and actual knowledge. We implemented the instrument in a higher education setting to assess its viability and usefulness and found that the instrument exhibited useful diagnostic capabilities and was able to identify common misconceptions among students. Although students performed well overall, there was a significant misunderstanding of how AI, especially GenAI systems, work. It also identified a lack of higher-level knowledge. The instrument is publicly available for use by others. We foresee its usefulness as a diagnostic that goes beyond understanding students'attitudes and perceptions of AI and GenAI use and tests multiple aspects of students'knowledge and conceptual understanding. This can enable the development of targeted instruction.
The growing presence of generative AI (GenAI) has raised questions about its role in supporting or hindering critical thinking in programming education. This study examined the influence of GenAI tools, particularly LLAMA, on students’ critical thinking skills during structured debugging tasks in a Java-based CS1 course. In a quasi-experimental design, 132 students were divided into an experimental group that received GenAI-assisted debugging instruction and a control group that received traditional instructor-led instruction. Both groups completed structured debugging tasks, a closed-book post-evaluation, and a program writing task, while students in the experimental group completed a survey. Students who used LLAMA for debugging tended to link lower- and higher-level thinking skills through stronger conditional associations between understanding code, applying solutions, and extending them into creative solutions, while students who used manual debugging showed stronger connections between analyzing problems and evaluating solutions. Permutation tests confirmed significant differences in skill associations between groups. However, these network-level differences did not always translate into higher performance scores on individual skills, as the control group achieved higher scores in core skills with lower-level cognitive demands. Creativity was also one of the differences observed between the groups. In the LLAMA group, Creativity was connected to Application, with a weaker connection to Analysis, whereas in the control group, it was disconnected from other skills. However, in some cases, students’ solutions in the control group showed structural changes that went beyond what was taught. The findings suggest that GenAI tools like LLAMA may be associated with different patterns of critical thinking during debugging. Yet, their association with individual skills seems limited in the absence of well-structured instructional support. To properly harness GenAI in debugging education, educators must adopt pedagogical approaches that guide students toward reflection, independent reasoning, and balanced cognitive engagement.
Yazid Albadarin, M. Saqr, N. Pope et al.· Technology, Knowledge and Le...· 0 citations
The findings suggest that the impact of GenAI use is present in various contexts, highlighting the need for instructional guidance on how students should use GenAI as a learning aid, and insights for other instructors that wish to integrate GenAI tools into computing curricula.
Valeria Ramirez Osorio, Ido Ben Haim, Ahmed Ashraf et al.· Annual Conference on Innovat...· 1 citation
This research paper describes an initiative to supplement novice design knowledge with generative AI (genAI). Past studies show students feel they don’t have sufficient genAI knowledge and skills, and faculty are concerned about students’ ability to evaluate genAI. Although there is concern that students may become too reliant on genAI, research shows that genAI can support the development of and ultimately automate aspects of the engineering design process. This research identifies students’ perceptions of the use of genAI in engineering design courses. Building on the designer patterns described in Crismond and Adam’s Informed Design Learning and Teaching Matrix, this work explores genAI’s impact on engineering design. Findings show students are generally comfortable using genAI, they feel they are using it effectively, that it increases their quality of work and efficiency, and feel it enhances their creativity. Students reported mixed messages from instructors and concerns about ethical use, unfair advantage, and over-reliance.
N. Nelson, C. Rennick, Silas Ifeanyi· Proceedings of the Canadian...· 0 citations