This vision reframes AI code review from automated commenting to human-AI sensemaking before integration, and outlines a research agenda for studying review conversations, designing conversational AI review capabilities, and evaluating their impact on software evolution and maintenance.
Abstract
AI-based code review tools increasingly promise to help developers inspect pull requests, identify defects, and improve code quality. Yet most current approaches frame code review as a one-shot commenting task: given a diff, the system produces warnings or suggestions. This framing overlooks a central property of modern code review: review is a conversation. Human reviewers do not merely comment on code; they ask questions, explain expectations, negotiate design trade-offs, request evidence, transfer project knowledge, document rationale, and collectively decide whether a change is good enough to integrate. In this vision paper, we argue for conversational AI review assistants: systems that participate in code review as interactive partners rather than static comment generators. Such assistants should identify when conversation is needed, ask grounded questions, respond to developer explanations, summarize unresolved issues, help capture rationale, and know when to abstain or escalate to human reviewers. Such a paradigm shift requires novel evaluation methodologies as well. We outline a research agenda for studying review conversations, designing conversational AI review capabilities, and evaluating their impact on software evolution and maintenance. Our vision reframes AI code review from automated commenting to human-AI sensemaking before integration.
AI coding agents are generating code at volumes that exceed the capacity of traditional peer review. At the same time, existing AI code review tools over-index on low-value suggestions such as style and best practices while under-indexing on the concerns human reviewers prioritize most: correctness, security, and performance. We present ARCTIC, an AI-powered Code Critique system that reframes code review around three capabilities: intent prediction, which infers why a change was made from conversation logs and metadata; drift detection, which measures divergence between the developer's intent and the agent's output via backtranslation; and code spotlight, which ranks the regions of a diff most warranting human scrutiny. We ground these capabilities in a six-theme taxonomy derived from 18,000 code reviews. Offline evaluation shows that intent prediction achieves 0.86 F1, drift detection reaches near-perfect ordinal agreement with human annotators (QWK = 0.907), and spotlight outperforms the baseline AI reviewer by 2.4x on quality estimation at 5x fewer tokens. In the experimental rollout, the drift scores reduces code misalignment by an additional 5.76 points (p = 0.026), intent prediction receives 90.2% approval, and zero defects have been attributed to self-reviewed diffs since launch.
C. Maddila, Mashrur Rashik, E. Khan et al.· 0 citations
AI coding assistants such as GitHub Copilot and Cursor have evolved from code-suggestion tools into conversational collaborators, enabling vibe-coding workflows in which developers guide AI-generated code through natural-language dialogue. Although researchers have increasingly recognized the importance of AI coding agents and begun examining their impact on open-source development, a comprehensive understanding of how developers'chat-based interactions with AI relate to subsequent open-source development and collaboration remains limited. This hinders efforts to effectively design, evaluate, and govern AI-assisted open-source software development. To address this gap, we collected 13,360 AI conversation sessions comprising 79,172 user messages from 1,356 OSS repositories, linked them to repository development histories, and complemented this analysis with a targeted developer survey. We find heavier AI use in smaller, less mature, and less collaborative repositories. After AI adoption, projects tended to show more active contributors and lower contributor concentration (p<.001), although communication remained highly concentrated. Code Writing was the dominant chat purpose, and nearly all AI chat sessions were followed by subsequent commits. We find no broad deterioration in code-quality signals or pull request merging rates. However, developers perceive others'AI-generated code as harder to maintain than their own (p = .029) and view AI as lowering barriers to OSS contribution. While most developers (68%) are willing to share their chat, concerns remain around appearing incompetent, increasing reviewer burden, and exposing ideas to competitors. These findings provide a large-scale empirical characterization of AI-assisted OSS contribution and offer practical insights for designing and governing responsible vibe-coding practices in open-source development.
Zihan Fang, Yueke Zhang, Ningzhi Tang et al.· 0 citations
Code review is a critical quality assurance practice in software engineering development, and AI coding agents are increasingly generating review comments on pull requests. However, little is known about how developers actually respond to such agent-generated feedback. In this paper, we present the first large-scale empirical study on the resolution of agent-generated code review comments. We analyze $54{,}791$ comments generated by five widely used coding agents (i.e., Copilot, Cursor, Codex, Devin, and Claude) across $342$ Python repositories on GitHub. We examine (1) resolution rates across agents and comment types, (2) the role of developer experience, and (3) characteristics that influence comment usefulness. Our results show that resolution rate varies considerably across agents, with Copilot accounting for the majority of resolved comments (72.9\%). Core developers resolve the majority of agent-generated feedback, particularly for \textit{design} and \textit{evolvability}-related comments, while peripheral developers are more involved in resolving \textit{functional defect} comments. Through open card sorting of 470 unresolved comment discussions, we identify \textit{ten} discussion patterns explaining why comments remain unresolved, with \textit{incorrect suggestions} and \textit{intentional design decisions} being the most prevalent. Finally, our analysis reveals that the presence of an inline \textit{code suggestion} is the strongest predictor of comment resolution, while lengthy and complex comments are less likely to be acted upon. Our findings provide insights for improving AI-generated code review feedback and its integration into development workflows.
Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy et al.· 0 citations
Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to understand. Developers struggle to assess the validity of LLM-generated reviews, making it difficult to gauge how much trust to place in them. The role of Explainable AI (XAI) in code review and its impact on trust remain underexplored. Objective: We study the influence of XAI on developer trust in AI-assisted code reviews. Method: We conducted a within-subjects user study with 34 participants, comparing three LLM-based code review systems with varying levels of XAI support: Condition A (detailed explanation and review feedback), Condition B (review feedback only), and Condition C (no explanations). Participants reviewed real-world code change requests alongside the AI-generated reviews. We measured trust perceptions, agreement with the AI recommendation, the reasoning given for each decision, and the time taken. Results: The level of explanation significantly influences both trust and agreement with AI recommendations, but in different ways. Full explanations (A) yield the highest perceived trust (M = 3.99/5) but not the highest agreement, whereas moderate explanations (B) achieve the highest agreement (89.22%). This could suggest that more explanation prompts developers to question AI recommendations more frequently. No explanations (C) results in the lowest trust and agreement. Explanation level did not significantly affect review time. The most commonly cited reasons for decisions were code readability and correctness. Conclusion: Incorporating XAI into code review significantly changes trust perceptions and agreement with AI recommendations. These results inform the design and evaluation of trustworthy AI-based code review systems, as well as studies on the human factors of AI-assisted software development.
Zhenhan Gao, Marvin Muñoz Barón, Umm E. Habiba et al.· 0 citations
Code review is essential for ensuring software quality and supporting collaboration, yet prior work shows that developers can interpret code review comments differently. These differences can hinder effective communication, particularly in collaborative settings. To address this challenge, we explore the potential of personified code review explanations. We report initial findings from an ongoing mixed-methods user study in which developers evaluated persona-aligned review comments across multiple code snippets. Our results suggest that preferences for explanation styles vary across problem-solving styles, experience levels, and roles. Across problem-solving style profiles, developers valued explanatory depth, learning support, practical suggestions, and risk awareness over conciseness, highlighting the need to balance personalization with clarity and trust. Based on these findings, we outline a vision for inclusive, human-centered AI-assisted code review systems that adapt feedback to developers'problem-solving preferences.
Shamse Tasnim Cynthia, Ratnadira Widyasari, Banani Roy et al.· 0 citations
The results show that agent-involved collaboration patterns, especially reviews initiated by AI agents or involving multiple AI agents, are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption, but these efficiency gains do not translate into better review quality.
Suzhen Zhong, Shayan Noei, Bram Adams et al.· 2 citations