The results show that AI effectively lowers cognitive barriers by summarizing pull requests, detecting minor localized errors, and helping developers discover system-level problems, but AI tools must adopt a "do more with less" design philosophy where the design should focus on high-reliability features and not overload the users with information.
Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to understand. Developers struggle to assess the validity of LLM-generated reviews, making it difficult to gauge how much trust to place in them. The role of Explainable AI (XAI) in code review and its impact on trust remain underexplored. Objective: We study the influence of XAI on developer trust in AI-assisted code reviews. Method: We conducted a within-subjects user study with 34 participants, comparing three LLM-based code review systems with varying levels of XAI support: Condition A (detailed explanation and review feedback), Condition B (review feedback only), and Condition C (no explanations). Participants reviewed real-world code change requests alongside the AI-generated reviews. We measured trust perceptions, agreement with the AI recommendation, the reasoning given for each decision, and the time taken. Results: The level of explanation significantly influences both trust and agreement with AI recommendations, but in different ways. Full explanations (A) yield the highest perceived trust (M = 3.99/5) but not the highest agreement, whereas moderate explanations (B) achieve the highest agreement (89.22%). This could suggest that more explanation prompts developers to question AI recommendations more frequently. No explanations (C) results in the lowest trust and agreement. Explanation level did not significantly affect review time. The most commonly cited reasons for decisions were code readability and correctness. Conclusion: Incorporating XAI into code review significantly changes trust perceptions and agreement with AI recommendations. These results inform the design and evaluation of trustworthy AI-based code review systems, as well as studies on the human factors of AI-assisted software development.
Zhenhan Gao, Marvin Muñoz Barón, Umm E. Habiba et al.· 0 citations
The results show that agent-involved collaboration patterns, especially reviews initiated by AI agents or involving multiple AI agents, are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption, but these efficiency gains do not translate into better review quality.
Suzhen Zhong, Shayan Noei, Bram Adams et al.· 2 citations
AI-assisted development tools enable software engineers to generate implementations at substantially higher speed and volume than in traditional workflows, yet relatively little is known about how existing guardrails evolve in response.
A student survey study is presented that examines perceptions of LLM output understanding, validation effort, trust and the perceived usefulness of vibe modeling across several AI-assisted development scenarios to inform future studies for trustworthy and explainable AI-based software engineering via vibe modeling.
Shalini Chakraborty, M. Mittermaier, J. Michael· 0 citations
UX heuristic evaluation is a core human factors method for assessing interface designs, but expert-led approaches are constrained by expert availability and time demands. Although multimodal large language models can automate heuristic evaluation from screenshots, full automation raises concerns about inaccurate design interpretation, unreliable reasoning, and limited transparency. This study proposes a structured human-in-the-loop architecture that reframes heuristic evaluation as a problem of cognitive labor distribution. Two AI agents—the Design Representation Generator and the Heuristic Evaluator—collaborate with a human supervisor. The supervisor reviews and corrects intermediate design representations and challenges, contextualizes, or refines AI-generated UX issues through correction and adjustment loops. Across four Android mobile app task scenarios, the human-in-the-loop system was compared with a fully automated baseline using the same two-agent pipeline without human supervision. The human-in-the-loop system produced heuristic evaluation results that showed substantially greater alignment with human experts than those generated by the automated baseline.
Hyeonyeong Lee, Dusan Baek, Woojin Park· Proceedings of the Human Fac...· 0 citations
Artificial intelligence (AI) is changing how educational institutions interact with donors, alumni and stakeholders. While early applications focused on automating routine tasks, the emergence of generative AI (GenAI) and large language models (LLMs) has introduced a new paradigm: AI as a strategic thought partner. This paper explores how GenAI can enhance leadership, creativity and decision making within university advancement, drawing on personal experience during a large-scale university merger and the applied use of multiple LLMs — including Microsoft Copilot, Claude and ChatGPT — as strategic tools. It presents principles of effective prompt design and use of ‘personas’ that enable advancement professionals to collaborate productively with AI and proposes a values-led advancement AI governance framework to support ethical and responsible use. The discussion highlights both opportunities and risks — emphasising the importance of human review, data privacy and the need for advancement-specific guidelines. The paper concludes that GenAI, when implemented with clear guardrails and institutional integrity, has the potential to elevate advancement from an operational function to a strategic, mission-aligned partner in shaping the future of higher education. This paper is also included in The Business & Management Collection, which can be accessed at https://hstalks.com/business/.
Liz Hawkins· Journal of Education Advance...· 0 citations