LLM-based Specification-Driven Test Oracle Enhancement

Abstract

Software testing critically depends on test oracles, yet existing test oracles are often incomplete and insufficient for detecting bugs where implementations deviate from their specifications. Meanwhile, despite advances in test oracle construction, existing techniques typically rely on coarse-grained failure signals or require substantial manual effort, and thus remain inadequate for detecting specification-violation bugs. To address these limitations, we propose JavaOracle, a specification-driven approach that leverages large language models (LLMs) to reason over specifications and systematically enhance test oracles. Specifically, JavaOracle consists of three stages. First, it leverages LLMs to analyze specifications and derive additional test oracles that are not covered by existing tests, while integrating a root-cause-guided repair workflow to ensure that the enhanced test cases are syntactically and semantically valid. Second, it designs a multi-agent debate workflow to distinguish previously unknown specification-violation bugs from assertion failures caused by LLM hallucinations, thereby mitigating the impact of hallucinations. Third, unlike existing approaches that stop at bug detection, JavaOracle further automates test case minimization and bug report generation, producing submission-ready reports without manual effort. We evaluate JavaOracle on 3,961 test cases for Java SE API from the OpenJDK. Experimental results on the latest OpenJDK standard library show that JavaOracle substantially outperforms state-of-the-art baselines, including Fuzz4All, ChatAssert, and Randoop. Cumulatively, JavaOracle discovers 45 previously unknown bugs, which have already been confirmed/fixed by developers, with many persisting since their initial implementation. In an entire pipeline running, JavaOracle reduces execution failures to an average of 6 reports per run, achieving 88.9% precision. In contrast, baseline approaches required exhaustive manual inspection to identify only 0, 3, and 2 real bugs, respectively. Further analysis shows that test cases enhanced by JavaOracle achieve high validity, with an execution pass rate of 79.1%, compared to 35.4%, 92.2% (55.5% test cases unchanged), and 77.6% for the baselines. Ablation studies further demonstrate the effectiveness of JavaOracle components, while the automated pipeline significantly reduces manual analysis effort.