CodeOwl: Automatic Generation of Tiered Parsons Problems for Introductory Programming

Comparative Evaluation of Large Language Models in Computer Programming Education

Delivering high-quality formative feedback in large introductory programming courses remains challenging despite the widespread use of automated grading systems. Although large language models (LLMs) can generate natural-language explanations grounded in student code and test outcomes, their diagnostic reliability and pedagogical suitability require systematic evaluation. This study presents a comparative analysis of six LLMs for generating formative feedback on introductory Java programs containing predefined defects under controlled conditions. Model outputs were evaluated across key dimensions of feedback quality and performance. Results reveal substantial cross-model variation, particularly in multi-defect scenarios. More precise fault localization was associated with higher diagnostic accuracy and completeness, but also with more frequent instructional-constraint violations. No single model consistently outperformed others across all criteria, and response latency did not reliably correspond to diagnostic quality. These findings underscore the context-dependent nature of LLM-generated formative programming feedback.

Assessing the effectiveness of large language models for generating and estimating time complexity of code segments

The time complexity of algorithms is a critical concept in computer science and engineering, and it is recognized as a fundamental topic in the Association for Computing Machinery (ACM) curricular recommendations. In alignment with these guidelines, numerous universities worldwide incorporate this topic into their introductory computer science courses, which typically have large student enrollments annually. Consequently, there is an increasing demand for automation in both instructional and assessment processes. This study explores the potential of Large Language Models (LLMs) to assist teaching staff in generating source code segments with predefined time complexity and determining the time complexity of given code segments, with applications in educational and examination contexts. We proposed a novel methodology for LLM evaluation in the aforementioned context and evaluated three prominent LLMs: ChatGPT, Gemini, and Llama, on their ability to generate and analyze C code segments exhibiting linear, logarithmic, quadratic, and exponential time complexities. A framework was developed to automate the prompt and segment generation and time complexity determination using two mainstream prompt engineering methods: zero-shot and chain-of-thought, and assessed the differences in code generation and time complexity analysis. A total of 960 generated segments were assessed on the correctness of time complexity, structural appropriateness, and suitability for exam use. The results suggest that ChatGPT is the most suitable LLM for generating segments with predefined time complexity (success rate goes up to 61%). All LLMs yielded the best results in generating linear segments, while exponential complexity posed the greatest challenge overall. A subset of generated segments was extracted to evaluate the time complexity determination capabilities. All three LLMs were asked to find the time complexity of each extracted segment. The most accurate LLM is ChatGPT (79.6%). We also assessed how good each LLM is in determining the time complexity of segments generated by itself. Llama outperforms others in that task (83% of successful determinations) when the zero-shot prompt method is used. The findings suggest that current LLMs cannot fully automate question generation and time complexity problem solving. However, they can substantially support the process and reduce the workload for educators.

Scaffolding Autocomplete: Improving Guidance for Learners using Generative Code Suggestions

Modern programming tools use generative AI (GenAI) to suggest code to the user as they type, interrupting their problem-solving behavior and undermining the development of their programming critical thinking skills. In this paper, we present a scaffolded programming exercise designed to support student differentiation between good and bad GenAI code suggestions based on negative expertise–that identifying why an answer is wrong is part of developing conceptual knowledge. We compare a version of the tool that showed one suggestion (correct or not), to a version that showed three suggestions (one of which was correct). We present results on performance and error rates as well as qualitative findings centered on Pintrich and DeGroot’s theory of self-regulation. Students reported that the single suggestion version better aligned with industry tools and presented a lower cognitive load. Students also reported that the multiple suggestion version caused them to slow down and think critically about the line under consideration, the overall purpose of the code, and the benefits of planning.

Integration Of Languagetool Api For Automated Spelling And Grammar Assessment In English Writing

English writing proficiency is a fundamental academic competency and a key indicator of students' language mastery in higher education. Theoretically, this complexity is grounded in the Complexity, Accuracy, and Fluency (CAF) framework, with the accuracy aspect specifically emphasizing adherence to grammatical and spelling rules. However, the manual assessment processes typically employed by instructors are often hindered by heavy technical workloads, high operational costs, and potential inter-rater inconsistency, ultimately limiting the frequency and depth of formative feedback provided to students. While commercial Automated Writing Evaluation (AWE) tools offer a partial solution, many operate as opaque "black box" systems reliant on proprietary third-party APIs, raising concerns regarding data reliability, institutional privacy, and a lack of pedagogical transparency for learners. This research seeks to address these issues by developing a transparent, self-hosted writing assessment prototype using the Design Science Research (DSR) framework. The system is built on the Laravel 12 framework—chosen for its modularity and security—and integrates the open-source LanguageTool API to provide rule-based feedback on spelling and writing style. The prototype's performance was rigorously evaluated against a "gold standard" established by independent expert raters using a dataset of authentic student essays. Validation results demonstrate high reliability, with the system achieving significant accuracy in spelling and grammar detection. Furthermore, the system exhibits high technical efficiency with rapid response times. This research aims to produce a digital solution capable of further development that bridges the gap between traditional assessment methods and modern educational technology; the solution is also expected to effectively alleviate the technical workload of instructors while empowering students to become more independent through real-time instructional feedback based on language usage standards.

From Repositories to Practice: LLM-Based Personalization in Programming Education

Investigating How Problem-Based Projects Can Shape Engineering Students' Use of Generative AI for Coding

Generative AI is rapidly transforming workflows in software development. In mechanical engineering education, software skills are taught as a supplemental tool for modeling, but students often lack the experience to leverage AI assistance to approach larger-scale, complex coding tasks. Following a problem-based learning design, 230 third-year students were given a cart-pole system and a series of milestones to achieve. With their choice of generative AI support, they wrote C and Python code to control and analyze data from the cart-pole. We investigated whether this programming challenge would encourage students to adopt programming-specific generative AI strategies, e.g. code editor embedded AI assistants, when presented with the option. Data collected from TAs and student questionnaires indicated that most students continued with default approaches of using chatbots for code generation and debugging. These results indicates that students would benefit from structured training in effective generative AI usage for software coding.