TestEvo-Bench is a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior.
Abstract
Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite. We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.
Three applications, leveraging on the mechanisms of Eiffel and design by contract, hold significant promise to address some of the challenges of program testing, software maintenance, and automatic program repair.
Li Huang, Bertrand Meyer, M. Oriol· Communications of the ACM· 0 citations
Mutation testing is an approach to checking the robustness of test suites. The program code is slightly modified by mutations to inject bugs, and a test suite is robust enough if it finds them. Mutation testing tools provide sets of mutation operators, such as swapping arithmetic operators, to make small modifications to the program. The results of mutation tests depend directly on the possible mutations. These mutations should cause actual changes in the program behavior, but also should not prevent the program from being loaded and executed. The more advanced mutations are, the more they challenge the test suite. Existing non-model-based mutation testing tools do not support the definition of advanced mutation operators that go beyond manipulating a small number of adjacent instructions within a single method. Thus, we present a
model-driven
approach where mutations of Java bytecode can be flexibly defined as model transformations. Our tool, Model-based Mutation Testing (MMT), implements this approach and includes model transformations for conventional and advanced mutation operators, such as deleting overridden methods or changing type casts. To evaluate the effectiveness and efficiency of model-driven mutation testing, we have applied MMT to all projects and versions in Defects4J, a well-established collection of real-world Java projects with reproducible bugs. We check for MMT’s ability to generate mutants close to real bugs and compare it with the non-model-based mutation testing tools Jumble, PIT, and
$$\mu $$
μ
BERT. Our evaluation shows that MMT and PIT are significantly more effective and efficient than
$$\mu $$
μ
BERT and Jumble. MMT even outperforms PIT in its ability to generate such realistic bugs, with similar efficiency per generated mutant. Jumble and
$$\mu $$
μ
BERT are one or two orders of magnitude slower than MMT and PIT. There are some bugs reconstructed by only one of the tools, including some that only the advanced operators of MMT could replicate. Fifteen percent of the Defects4J project versions had bugs that could not be reconstructed by the mutation operators of any of the investigated tools. This shows that further research in mutation operators, as enabled by MMT, has high potential. Our mutation testing tool MMT is available online https://gitlab.uni-marburg.de/fb12/plt/modbeam-mt/mmt, as well as all the evaluation data (Ancona et al., Evaluation data of comparison of mutation testing tools. https://doi.org/10.5281/zenodo.20054492).
Freya Ancona, Christoph Bockisch, Daniel Neufeld et al.· Journal of Software and Syst...· 0 citations
This paper introduces and empirically study the phenomenon of error propagation, where faults in generated code are systematically replicated in associated test artifacts, and examines whether LLM-generated code biases the generation of subsequent tests.
Michael Konstantinou, Florian Tambon, Mike Papadakis· 1 citation
The experimental results show that the Cypress-based end-to-end test suite has short and stable execution times, and resilient data-cy attributes significantly reduce maintenance overhead when UI changes occur.
Quoc-Binh Nguyen, Truc-Ly Phan Nguyen, Ngoc Hong Tran et al.· 0 citations
Large Language Models (LLMs) have demonstrated remarkable potential in automated code generation. However, existing test-driven code generation and refinement frameworks are often hindered by the tests' quality: they typically treat self-generated tests as ground truth, leading to ineffective debugging loops where code is modified to satisfy erroneous tests. To address this, we propose BiVCoder, a diagnosis-driven multi-agent framework featuring a novel bidirectional code-test diagnosis mechanism. BiVCoder coordinates three specialized agents—Coding Agent, Test Agent, and Review Agent. Central to this architecture is the Review Agent, which serves as a diagnosis and decision-making hub. By integrating an MCP-based code-test execution tool, the Review Agent rigorously executes programs to diagnose failure root causes, distinguishing between implementation bugs and test case deficiencies, and subsequently triggers targeted repairs (Coding Agent or Test Agent). Furthermore, we introduce BiVCoder-SFT, a role-specific instruction fine-tuning scheme. We construct high-quality datasets to fine-tune the Qwen3-4B base model into specialized agents for coding, testing, and reviewing. Extensive experiments on HumanEval, MBPP, and their rigorous ''ET'' variants demonstrate the superiority of our approach. With GPT-3.5, BiVCoder achieves a Pass@1 of 77.3%. Notably, the specialized BiVCoder-SFT achieves an average score of 79.5% across the four datasets, not only outperforming its base model (69.7%) but also surpassing larger general-purpose models such as Qwen3-8B (77.8%). Additionally, BiVCoder boosts the performance of more powerful models, improving DeepSeek-V3.2 from 87.2% to 92.1% on HumanEval.
Xiaoyang Li, Jinhao Dong, Wenhang Shi et al.· Proceedings of the 32nd ACM...· 0 citations
TestAgent, a multi-agent tool implemented as a VS Code extension that automates the generation of high-quality unit tests for Java projects using repository-level Code Knowledge Graphs, demonstrates its practical utility for regression testing and bug discovery.
Ye Shang, Quanjun Zhang, Zheng Zhan et al.· SIGSOFT FSE Companion· 0 citations