A method called Personalized Probing Puzzles to evaluate students' understanding of their own code, and a pilot study shows that it can help identify potential gaps in students' understanding of their own code.
Abstract
The rapid development and popularization of AI-enabled coding agents have meant software engineering students and professionals cannot be assumed to understand their own code, which risks academic integrity and professional accountability. We developed a method called Personalized Probing Puzzles ($p^3$) to evaluate students'understanding of their own code, and tested $p^3$ in a graduate-level cloud computing course. Our pilot study shows that $p^3$ can help identify potential gaps in students'understanding of their own code. The puzzles are automatically generated, asynchronously administered, and finished in minutes. Future work is needed to correlate puzzle results with code understanding and to embed $p^3$ in a professional code review process.
Generative AI is increasingly permeating software engineering, enabling developers to generate functions, files, and even entire applications from natural language specifications. AI systems are also becoming more personalized, adapting outputs based on inferred user characteristics and interaction history. While personalization may improve the development experience, it raises concerns that generated software could be shaped by attributes of the developer rather than by task requirements alone. Prior work has shown that generative AI can produce biased software artifacts, but little is known about how developer identity can bias generated code. We characterize three dimensions through which inferred developer attributes can influence generated artifacts: interface design, template content, and code structure. First, through controlled experiments on 800 AI-generated websites, we find that age- and gender-related signals produce significant differences across all three dimensions. Second, we conduct an observational study and follow-up interviews with 20 participants who used AI to create a personal website to both examine how personalization impacts software artifacts in practice, and also to understand how programmers perceive the boundary between personalization and bias. Together, our results show that developer attributes can meaningfully influence generated software beyond stated requirements, highlighting a previously underexplored tension between personalization and fairness in AI-assisted programming.
Code review is essential for ensuring software quality and supporting collaboration, yet prior work shows that developers can interpret code review comments differently. These differences can hinder effective communication, particularly in collaborative settings. To address this challenge, we explore the potential of personified code review explanations. We report initial findings from an ongoing mixed-methods user study in which developers evaluated persona-aligned review comments across multiple code snippets. Our results suggest that preferences for explanation styles vary across problem-solving styles, experience levels, and roles. Across problem-solving style profiles, developers valued explanatory depth, learning support, practical suggestions, and risk awareness over conciseness, highlighting the need to balance personalization with clarity and trust. Based on these findings, we outline a vision for inclusive, human-centered AI-assisted code review systems that adapt feedback to developers'problem-solving preferences.
Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy et al.· 0 citations
Code comprehension is one of the most time-consuming tasks in software engineering, yet most LLM-based assistants produce explanations that ignore who is asking and force developers into a disruptive copy-paste workflow. We present TARS, an LLM-powered agent integrated into Visual Studio Code that supports program comprehension through autonomous explanations anchored directly to the code under analysis. Built around a lightweight Theory of Mind paradigm, TARS profiles a developer's expertise, role, and stylistic preferences, then adapts the depth and tone of its explanations accordingly, grounding them in project documentation via Retrieval-Augmented Generation. To evaluate TARS, we conducted a controlled experiment with 18 participants on non-trivial Java snippets. Participants using TARS completed tasks 26\% faster, reported lower cognitive load, and found the explanations meaningfully adapted to their profiles.
Leopoldo Todisco, Antonio Della Porta, Stefano Lambiase et al.· 0 citations
The rapid advancement of LLMs has opened new opportunities in automated software engineering, driving progress in code understanding, agent-based workflows, and productivity tools. However, existing code intelligence systems have largely sidelined the end-users they aim to serve—the developers themselves. Developers exhibit substantial heterogeneity across multiple dimensions: coding style, toolchain preferences, domain-specific expertise, and problem-solving strategies. Failing to account for these individual differences directly compromises both the effectiveness of code intelligence and the likelihood of its adoption. For example, a senior architect and a junior engineer ask: "Describe the authorization module." Without personalized context, the system produces a uniform response—verbose for the expert, incomprehensible for the novice. This gap motivates a fundamental shift: from one-size-fits-all to one-size-fits-one code intelligence. A developer's dynamic in-IDE behaviors—code authoring patterns, navigation pathways, debugging trajectories—implicitly encode a rich representation of their competencies and habits. If captured and interpreted systematically, these signals can enable Personalized Code Intelligence, formalized as: [EQUATION] where P is the developer persona derived from IDE behaviors, injected alongside code context C and instruction ℐ.
Yuhong Liu, Yu Su, Zhipeng Peng et al.· SIGSOFT FSE Companion· 1 citation
This research aims to bridge the gap by providing a comprehensive evaluation methodology for LLM-powered agents that is grounded in real-world software development practice and focuses on contamination-awareness, in thewild agentic behavior assessment, and trajectory-aware benchmarks and metrics.
The ever-increasing spread of AI-assistance tools into programming workflow presents a pedagogical challenge: assessing whether students possess genuine understanding of code they may not have written themselves. This paper proposes integrating video 'vivas' into programming assessments as a mechanism for validating code comprehension. In this approach, students submit code and recorded explanations in which they discuss their design choices and the logical flow of their code. Although this mechanism cannot fully eliminate misconduct, it reduces the feasibility of submitting AI-generated solutions without engaging with the code. We discuss implementation considerations, marking rubric development aspects, and consider the broader implications of this approach.
Filippos Pantekis, O. Petrovska· Annual Conference on Innovat...· 0 citations