Aug 2026· International Conference on Automated Software Engineering· Vol 33· 0 citations· 55 references
TL;DR
Results support a focused conclusion: LLM-generated review is most useful as complementary semantic guidance when paired with deployment-oriented test selection, rather than as a standalone testing artifact.
Large language models (LLMs) have opened new opportunities for unit test generation, but executable tests do not necessarily reveal real defects. This paper studies how historical real-bug mechanisms can be transformed into executable feedback targets for LLM-based unit test generation. The proposed framework constructs structural and semantic representations of real-bug records, retrieves mechanisms applicable to a focal method, and instantiates them as synthetic bugs that guide iterative test enhancement. We evaluate the approach on method-level real-bug detection tasks from Defects4J and show that mechanism-guided synthetic-bug feedback improves real-bug detection over execution-, coverage-, mutation-, knowledge-, and search-based baselines. The results suggest that organizing real-bug mechanisms as retrievable and executable feedback targets is an effective way to guide generated tests toward bug-triggering inputs and behavioral oracles.
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
Recent work argues that multi-agent LLM frameworks generate better unit tests than a single well-crafted prompt by iteratively refining coverage and cross-checking oracles. We test this claim directly on a 25-class Java system, comparing CANDOR, a multi-agent framework with JaCoCodriven coverage feedback and panel-based oracle review, against one-shot prompt test generation that submits to an LLM every source file and requirement of our example system in one request. We ran both strategies on GPT-4o and Gemini 2.5 Flash, and every resulting test suite is scored against a human-written baseline using PiTest (with 206 mutants). One-shot prompt outperforms CANDOR across the 25 classes combined: one-shot GPT-4o reaches a 69.90% mutation score versus CANDOR's 45.63%, and one-shot Gemini 2.5 Flash reaches 88.35% versus CANDOR's 35.92%. The one-shot prompt strategy ran on Gemini also surpasses the human baseline (83.98%) in about 45 seconds, while CANDOR with Gemini took roughly 50 hours. CANDOR's central problem is its Initialize stage: it considers the source of each class by itself. Consequently, it cannot reliably construct mocked dependency objects and can fail to produce tests that compile for classes with inter-class dependencies. Restricted to the classes it can handle, CANDOR's per-class mutation scores frequently match or beat the human baseline. Hence, CANDOR's weakness is architectural, not generative.
These findings suggest that while AI agents excel at rigorous boundary testing, they lack the “environmental awareness” needed to write stable, hermetic tests.
Preet Jhanglani, Zeel Desai, Vidhi Kansara et al.· Annual International Compute...· 0 citations
JavaOracle is proposed, a specification-driven approach that leverages large language models to reason over specifications and systematically enhance test oracles and further automates test case minimization and bug report generation, producing submission-ready reports without manual effort.
Ruifeng Fu, Yingquan Zhao, Meng Wang et al.· SIGSOFT FSE Companion· 0 citations
TATG introduces a unified objective representation that captures testing requirements derived from static analysis and dynamic feedback that enables fine-grained tracking of satisfied and unresolved requirements throughout generation.
Guancheng Wang, Qinghua Xu, Lionel C. Briand· 0 citations