TDRepro: A Neurosymbolic Approach to Reproducing Timing-Dependent Flaky Test Failures

Abstract

Tests that nondeterministically pass or fail on the same version of code are known as flaky tests, and their failures substantially hinder regression testing and debugging. Reproducing failures from timing-dependent (TD) flaky tests is particularly challenging because these failures depend on rare thread interleavings that may not occur even after running the test many times. Prior approaches reproduce such failures by searching for locations at which to inject delays during execution, but this search process can be expensive. We present TDRepro, an automated approach that leverages large-language models (LLMs) and program analysis to efficiently reproduce TD-test failures. TDRepro first identifies relevant methods by collecting executed methods during the test run and identifying those that execute concurrently with other methods. It then ranks these methods by computing the embedding similarity between each method’s code and the observed failure log. Finally, TDRepro uses an LLM to identify candidate locations for delay injection within the top-ranked methods. TDRepro validates these candidates by injecting delays and rerunning the test, iteratively refining the candidate locations based on unsuccessful attempts until the target failure is reproduced or the maximum number of iterations is reached. We evaluate TDRepro on 82 TD tests across 25 modules from open-source Java Maven projects. TDRepro reproduces 100% more failures than the prior state-of-the-art approach, FlakeRake, and achieves more reliable reproduction, with 55 tests reaching a failure reproduction rate of >99%, compared with 30 tests for FlakeRake. Repeatedly reproducing failures with TDRepro requires 1482.1× and 10.4× less time and costs 888.4× and 9.6× less than repeatedly rerunning tests and using FlakeRake, respectively.