The results demonstrate that EXCODER provides an effective, though imperfect, solution to this problem in automated code generation, offering developers the first way to implement ERC following test-driven development.
Abstract
Exception Related Code (ERC), which includes throw statements, conditions (if statements) that guard those throw statements, and try/catch blocks, is an essential component of software systems, allowing developers to detect and handle exceptional states that deviate from the expected program behavior. However, manually writing ERC across large codebases is tedious. We propose a novel task: retrofitting existing code with ERC. Namely, given code (without ERC) and Exceptional Behavior Tests (EBTs) (e.g., check if method throws InvalidArgumentException if null is given as the value to the argument) we aim to automatically generate missing ERC, such that the given tests pass. We design and implement Exception Coder (EXCODER) that performs context engineering to help Large Language Models (LLMs) tackle this task. EXCODER integrates static and dynamic program analysis with LLMs by providing the extracted contextual information to the LLMs. To evaluate EXCODER, we build a benchmark constructed from GitHub Java repositories, where we systematically remove ERC in 304 methods from 75 projects. Our results demonstrate that EXCODER provides an effective, though imperfect, solution to this problem in automated code generation, offering developers the first way to implement ERC following test-driven development. When combined with Qwen 2.5 Coder 32b, EXCODER achieves pass@1, 5, and 10 rates of 85.92% (12.56 percentage points over baseline), 86.18% (12.82 p.p. over baseline), and 86.51% (13.15 p.p. over baseline), respectively, on developer-written test suites. Our manual inspection of the generated code further reveals limitations of EXCODER, pointing to directions for future work.
The goal is to understand the code generation errors of foundation LLMs and explore the solution to resolve directly fixable errors, and to design and evaluate the LlmFix fixing method and constructed the LlmErrorEval dataset.
Hao Wen, Yue-Heng Zhu, Chao Liu et al.· Empirical Software Engineeri...· 0 citations
SWE-Flux is introduced, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs.
Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband et al.· arXiv.org· 1 citation
This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests to ensure that the results are rigorous.
Zeyad Farooq Lutfi· Al-Noor Journal of Engineeri...· 0 citations
Investigating Behavior Trees (BTs) as an alternative intermediate representation for LLM-based vulnerability detection suggests that BTs can provide a compact and useful structured representation for vulnerability detection with quantized, locally deployable LLMs.
Enna Bašić, A. Giaretta· ACM Workshop on Artificial I...· 0 citations
Static analyzers reason about program behavior without execution and report issues when code violates predefined rules. A key limitation is their production of inaccurate, incomplete results, generating spurious warnings and missing critical issues. Metamorphic testing is the most commonly used method to efficiently de...
Mei-Lin Li, Kai-Xuan Li, Zi-Fan Xie et al.· Proceedings of the 41st IEEE...· 0 citations
This work presents the development of an automatic refactoring tool for Java code built on top of the Eclipse JDT API. The proposed approach transforms control structures containing break and continue statements within different types of loops into semantically equivalent constructs that avoid their explicit use. To ac...
Andrés Juárez, J. Chicano, Rubén Saborido· arXiv.org· 0 citations