Jul 2026· International Conference on Artificial Intelligence Testing· pp. 76-83· 0 citations· 29 references
Abstract
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
Enterprise regression testing for microservice systems is often constrained by incomplete or outdated documentation. In practice, QA engineers frequently rely on real execution traffic to reconstruct business scenarios; however, turning raw traffic into replayable regression tests with stable validation logic remains labor-intensive and error-prone. This paper presents NL2Test, an end-to-end approach and tool that generates executable API regression tests from (i) a natural-language scenario description and (ii) a traffic capture recorded while executing the scenario. NL2Test addresses two coupled tasks: test case carving, which extracts a minimal replayable request sequence and reconstructs data dependencies so that dynamic values are bound from their responses rather than hard-coded; and assertion generation, which produces assertions aligned with business intent while avoiding non-deterministic fields and hallucinated paths. To improve reliability, NL2Test uses LLMs for semantic interpretation and constrained code synthesis, and uses deterministic algorithms for request filtering, dependency confirmation via value consistency, and assertion-path validation. We evaluate NL2Test on 51 industrial regression scenarios extracted from a large consumer-facing Internet company. NL2Test achieves an exact-match rate of 82.4% (42/51), and produces a functionally usable draft in 98.0% (50/51) of scenarios when allowing minor post-edits. In a 9-month production deployment starting in March 2025, NL2Test generated 3,196 test cases with an overall code adoption rate of 85.4%. These results indicate that traffic-grounded generation with deterministic guardrails can substantially reduce manual effort while improving regression automation in complex microservice environments.
Haozhen You, Zhen Dong, Jingjing Wang et al.· 0 citations
Experimental results indicate that the dynamic validation mechanism and minimal target repair strategy can reduce invalid generation while improving the executability, assertion effectiveness, and fault-revealing capability of generated tests.
Xiaorong Zhang, Na Wang, Chunyang Su et al.· International Conference on...· 0 citations
While Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, their effectiveness in unit testing is often constrained by insufficient context regarding external dependencies. This limitation is particularly pronounced in industrial settings, where proprietary code remains opaque to the model. To address this challenge, we present a systematic empirical study of multiple strategies for context enrichment and optimization in LLM‐based unit test generation, conducted on seven diverse projects (three open‐source and four proprietary industrial systems), encompassing 261 distinct methods. By evaluating seven implementations (ranging from basic prompts to optimized context reduction strategies) across 10 independent runs, we analysed a total of 28,710 test suites. Our results demonstrate that combining prompt engineering with external dependency retrieval achieves an average branch coverage increase of 11.52 percentage points on industrial software over the baseline, with statistically significant improvements across all competing implementations. Beyond coverage, richer context substantially reduces generation‐repair iterations, cutting median execution time by 51.3% in industrial projects. We further show that reducing external dependencies to method signatures alone decreases input token consumption by up to 46.6% (25.4% in industrial projects) while fully preserving the coverage and efficiency gains of the complete retrieval approach. To confirm that these benefits are not tied to a specific model, we replicate the core comparison across three LLM backends from different families, obtaining a consistent, statistically significant coverage improvement on industrial code in every case. These findings establish this optimized context strategy as a cost‐effective solution for scalable, industrial‐grade automated test generation.
Javier Ferrer, Francisco Chicano· Expert systems· 0 citations
This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.
Wenqiang Luo, J. Keung, Xiaoyu Shi et al.· 0 citations
This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests to ensure that the results are rigorous.
Zeyad Farooq Lutfi· Al-Noor Journal of Engineeri...· 0 citations