This work proposes ExplainBench, a benchmark to automatically evaluate explanations from coding agents, based on the intuition that informative explanations should enable an LLM to correctly answer questions, allowing quantitative comparison of explanation quality between agents.
Abstract
Large Language Model (LLM) agents have seen rapid adoption in software engineering. As agents take a greater role in the actual generation of code, they are making larger changes, spanning tens to hundreds of lines. This makes manual review of agent results increasingly infeasible, leading developers to turn to explanations to understand enacted changes. Despite this, there are no benchmarks that evaluate the trustworthiness of agent-generated explanations. To bridge this gap, we propose ExplainBench, a benchmark to automatically evaluate explanations from coding agents. ExplainBench is based on the intuition that informative explanations should enable an LLM to correctly answer questions, allowing quantitative comparison of explanation quality between agents. With this observation, we construct a suite of questions that evaluates whether explanations accurately describe (1) the intended behavior of buggy code and (2) the effect of applying the agent patch itself. Experiments first reveal that explanation quality is a distinct axis of agent evaluation: ExplainBench ranks agents differently from the widely-used SWE-bench Verified benchmark. A deeper breakdown of explanation quality in agents shows frequent problems in explanations, such that explanations often claim that a patch is correct when it is not. Based on this insight, we implement and evaluate an explanation audit agent which runs additional tests to validate and refine explanations. This agent improved the explanations of all evaluated agents, demonstrating agent explanations can be automatically made more trustworthy.
The AgentCodeReview system is presented, a multi-agent system that is able to conduct explainable code review and automated bug repair by leveraging software engineering agents with different code review tasks and its utility and extensibility to the field of explainable AI in software quality assurance are demonstrated.
B. N, T. L. Manasa· International journal of com...· 0 citations
It is shown that test suites generated by the spec-driven agent are superior to the baseline and human-authored tests in 77.8% and 56.7% of the cases, respectively, and demonstrated improvements on following best practices, readability, and edge-case coverage.
Michele Tufano, James McClure, José Cambronero et al.· 0 citations
Automated program repair (APR) agents are transitioning from research benchmarks to developer workflows, yet they still begin with bug reports written for human developers. While decades of research have established what makes a good bug report for humans (e.g., steps to reproduce, stack traces), it remains unclear whether these features transfer to LLM-based agents. We study this question in two analyses. First, we use statistical modeling to examine associations between 27 bug-report features and repair success across 433 SWE-bench Verified issues attempted by 87 repair agents. We find that fix suggestions, reproduction scripts, repository source code, and localization info are associated with higher resolution likelihood, while longer reports are associated with lower odds. Second, we conduct controlled ablations across 2 models and 17 problem-statement mutations on SWE-bench Pro, varying the information available to an agent while holding the underlying task fixed. We remove or isolate selected bug-report content, delete fault-localization cues, and test structural changes that flatten lists or remove section headers. We find that both models depend on localization cues and expected behavior, and that structural changes alone can reduce solve rates, even without removing any content. The two models diverge in how they handle missing information: Qwen searches more widely and can exhaust its turn budget, while Gemma commits to a plausible interpretation early and patches on it. Our findings indicate that a good bug report for an agent overlaps with, but is not identical to, a good report for a human: agents benefit most from concrete, executable, and well-localized information, whereas some qualities long emphasized for human readers, such as natural language steps to reproduce and readable descriptions, contribute little or even correlate with lower success.
Lara Khatib, N. Mathews, M. Nagappan et al.· 2 citations· ⚡1
The first empirical study focused on agent-reactive (AR) bugs is conducted, constructing a two-axis taxonomy covering observable symptoms and the LLM behaviors that trigger them and highlights challenges specific to LLM agents.
Jingyi Chen, Songqiang Chen, Hengcheng Zhu et al.· 0 citations
Large language models (LLMs) have shown exceptional capabilities in code understanding and generation. However, they still face significant challenges in analyzing and debugging code. Most existing works rely on a single model, which often struggles to detect and fix bugs in complex program structures and semantic logic. This paper presents DoTA, a novel debugging framework that enhances LLMs' debugging capabilities through multiagent collaboration. Our key innovation is two-fold. First, we enhance code understanding through automated hierarchical documentation analysis, enabling more effective bug detection and localization based on comprehensive program context. Second, we leverage a delta-of-thoughts process where multi LLM agents analyze different aspects of program correctness and iteratively contribute complementary insights to identify bugs. Our experiments show that combining different LLM based agents with enriched documentation context significantly improves LLMs' debugging capabilities. DoTA has been extensively evaluated on the Debug- Bench dataset of 4,253 debugging instances. It achieves an average improvement of 15% in bug detection accuracy across languages compared to GPT-3.5 with task background prompting. The framework shows particular strength in handling complex logical errors (+18.3%) and multiple bugs (+18.9%). On open-source models, DoTA enhances bug detection capabilities by 10.4–13.5%.