Jul 2026· ACM Transactions on Software Engineering and Methodology· 0 citations· 59 references
TL;DR
Test4Py is presented, a novel framework that enhances type correctness in automated test generation for Python by leveraging the program’s call graph to capture richer contextual information about parameters, and introducing a behavior-based type inference mechanism that accurately infers parameter types and constructs valid test inputs.
Abstract
Automated test generation has been extensively explored, yet generating high-quality tests for Python programs remains particularly challenging. Because of Python’s dynamic typing features, existing approaches, ranging from search-based software testing (SBST) to recent LLM-driven techniques, are often prone to type errors. Hence, existing methods often generate invalid inputs and semantically inconsistent test cases, which ultimately undermine their practical effectiveness. To address these limitations, we present Test4Py, a novel framework that enhances type correctness in automated test generation for Python. Test4Py leverages the program’s call graph to capture richer contextual information about parameters, and introduces a behavior-based type inference mechanism that accurately infers parameter types and constructs valid test inputs. Beyond input construction, Test4Py integrates an iterative repair procedure that progressively refines generated test cases to improve coverage. In an evaluation on 183 real-world Python modules, Test4Py achieved an average line coverage of 83.0% and branch coverage of 70.8%, outperforming state-of-the-art tools by 7.2% and 8.4% in relative gains, respectively.
Unit testing is crucial for software reliability in JavaScript, yet constructing comprehensive suites remains costly and error-prone. JavaScript’s dynamic features, flexible typing, asynchronous execution, and reliance on third-party libraries, pose persistent challenges for automated test generation. Although large language models (LLMs) show promise in code reasoning and synthesis, existing methods often miss the multilayer contextual information required for executable, semantically correct tests. This paper introduces JSTestCraft, an agentic, multi-layer framework for adaptive JavaScript unit test generation. JSTestCraft reconstructs missing context via three enrichment agents: library, structural, and semantic. These agents capture inter-function topology, third-party API semantics, and inferred type constraints in a shared contextual memory for reasoning-driven test synthesis. A testing and optimization layer generates, executes, and iteratively refines test cases. Evaluated on 20 real-world Node.js repositories, JSTestCraft outperforms state-of-the-art baselines, improving test pass rate by 60.9%, statement coverage by 14.2%, and branch coverage by 47.3%. Ablation analysis confirms each agent’s contribution to contextual completeness. Beyond metrics, JSTestCraft discovers 13 previously unknown bugs across 5 repositories, 6 of which received substantive acknowledgment from maintainers or community contributors. These results demonstrate that context reconstruction and agentic collaboration enable LLMs to perform more reliable and adaptive testing in dynamic JavaScript environments.
Test oracles determine whether a program execution is correct for a given input. Two common forms are assertion oracles, which compare observed outputs with expected results, and exception oracles, which verify that a program raises an expected exception. Automated test oracle generation (TOG) aims to reduce the manual effort involved in constructing such oracles. Although recent TOG methods, especially LLM-based approaches, have made rapid progress, their evaluation remains constrained by benchmarks that rely on automatically generated tests, narrow single-assert formulations, simplified developer-written tests, or limited oracle diversity. To address these limitations, we introduce OE25𝑑𝑒𝑣 , a multi-variant dataset curated from developer-written unit tests across 25 open-source Java projects spanning 56 modules, and TOGBench, an end-to-end benchmark suite for TOG. OE25𝑑𝑒𝑣 captures six oracle categories and preserves realistic settings, including single- and multi-oracle configurations, mixed assertion-and-exception oracles, and developer-authored custom oracles. TOGBench supports end-to-end experimentation by reintegrating generated oracles into runnable test suites and evaluating them via compilation, execution, false-positive analysis, and mutation testing. Our evaluation further shows that OE25𝑑𝑒𝑣 preserves substantially greater structural complexity than prior benchmarks and exposes marked performance degradation of representative TOG models on developer-written tests, particularly for assertion oracles.
Tasfia Tasnim, Matthew B. Dwyer, Soneya Binta Hossain· AIware· 1 citation
This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests to ensure that the results are rigorous.
Zeyad Farooq Lutfi· Al-Noor Journal of Engineeri...· 0 citations
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
These findings provide the largest empirical characterization of PBT in Python to date, highlight developers’ difficulties in adopting the technique, and expose limitations of current tool support.
Isadora Campos De Oliveira, Arthur Lisboa Corgozinho, Henrique Rocha et al.· Empirical Software Engineeri...· 0 citations
JavaOracle is proposed, a specification-driven approach that leverages large language models to reason over specifications and systematically enhance test oracles and further automates test case minimization and bug report generation, producing submission-ready reports without manual effort.
Ruifeng Fu, Yingquan Zhao, Meng Wang et al.· SIGSOFT FSE Companion· 0 citations