Future TOG systems should be evaluated not only by whether they predict the correct oracle type, but also by whether their predictions are grounded in meaningful exception-triggering evidence, to challenge the assumption that strong exception-oracle accuracy reflects robust use of exception semantics.
Abstract
LLM-based test oracle generation (TOG) methods report high accuracy on exception oracle generation, but it remains unclear what evidence drives these predictions. In particular, do models use explicit exceptional-behavior documentation such as Javadoc @throws clauses, or do they rely on recurring patterns in tests, code, and documentation? We investigate this question through a large-scale intervention-based study of three TOG systems spanning classifier-based and generative architectures and model sizes from roughly 110M to 7B parameters, evaluated on three real-world benchmarks comprising two generated-test datasets and a new benchmark of developer-written tests. We first remove Javadoc @throws clauses and find that accuracy changes only marginally, with the largest drop below one percentage point. This indicates that structured exception documentation is not the primary driver of exception-oracle prediction. We then apply attribution-guided substitution ablations to identify the signals that predictions depend on. The results show that high accuracy can be driven by shortcut signals: some models are highly sensitive to a small number of structural tokens, while others distribute reliance across many lexical cues. These findings challenge the assumption that strong exception-oracle accuracy reflects robust use of exception semantics. Future TOG systems should therefore be evaluated not only by whether they predict the correct oracle type, but also by whether their predictions are grounded in meaningful exception-triggering evidence.
This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests to ensure that the results are rigorous.
Zeyad Farooq Lutfi· Al-Noor Journal of Engineeri...· 0 citations
The oracle problem (determining the correct expected outcome for a test) remains a major bottleneck in automated testing, and is increasingly relevant as non-experts rely on AI-generated code they cannot reliably validate. We study whether large language models (LLMs) can generate generalizable test oracles directly from natural-language business requirements, without access to source code or example input-output pairs. We propose a reproducible, requirement-driven pipeline grounded in Defects4J. For each of 10 real bugs from Defects4J Lang (Bugs 1 and 3-11), we (i) extract behavioral changes via buggy/fixed diffs, (ii) manually translate the change into a business requirement, (iii) construct a requirement-derived oracle (REQ) as a gold standard, and (iv) prompt five LLMs (DeepSeek-V3, Gemma-3n, Llama-3, Mistral-7B, and Qwen-3) to generate Java oracle code. We evaluate oracle correctness and generalization under two targets: agreement with REQ and agreement with the system under test (SUT), reporting macro-averaged accuracy, precision, recall, and F1. LLMs achieve non-trivial generalization but with substantial bug- and model-level variance. Generated oracles align more closely with REQ than with SUT, and correlations between requirement technicality/ambiguity ratings and oracle accuracy are weak with wide confidence intervals. No detectable linear relationship exists between requirement properties and oracle accuracy in this dataset, suggesting that pretraining coverage and the semantic specificity of the required behavior dominate oracle correctness. As a pilot proof of concept, these findings are preliminary and are intended to establish feasibility and motivate larger-scale empirical investigation.
Test oracles determine whether a program execution is correct for a given input. Two common forms are assertion oracles, which compare observed outputs with expected results, and exception oracles, which verify that a program raises an expected exception. Automated test oracle generation (TOG) aims to reduce the manual effort involved in constructing such oracles. Although recent TOG methods, especially LLM-based approaches, have made rapid progress, their evaluation remains constrained by benchmarks that rely on automatically generated tests, narrow single-assert formulations, simplified developer-written tests, or limited oracle diversity. To address these limitations, we introduce OE25𝑑𝑒𝑣 , a multi-variant dataset curated from developer-written unit tests across 25 open-source Java projects spanning 56 modules, and TOGBench, an end-to-end benchmark suite for TOG. OE25𝑑𝑒𝑣 captures six oracle categories and preserves realistic settings, including single- and multi-oracle configurations, mixed assertion-and-exception oracles, and developer-authored custom oracles. TOGBench supports end-to-end experimentation by reintegrating generated oracles into runnable test suites and evaluating them via compilation, execution, false-positive analysis, and mutation testing. Our evaluation further shows that OE25𝑑𝑒𝑣 preserves substantially greater structural complexity than prior benchmarks and exposes marked performance degradation of representative TOG models on developer-written tests, particularly for assertion oracles.
Tasfia Tasnim, Matthew B. Dwyer, Soneya Binta Hossain· AIware· 1 citation
This paper introduces and empirically study the phenomenon of error propagation, where faults in generated code are systematically replicated in associated test artifacts, and examines whether LLM-generated code biases the generation of subsequent tests.
Michael Konstantinou, Florian Tambon, Mike Papadakis· 1 citation
This work introduces a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance, and proposes DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures.
Amal Akli, Melissa Akli, Cedric Richter et al.· 0 citations
This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.
Wenqiang Luo, J. Keung, Xiaoyu Shi et al.· 0 citations