Systematic review of test case generation strategies for deep learning systems

Abstract

Deep learning (DL) systems are increasingly used in safety- and mission-critical domains, yet their testing remains difficult because conventional software testing techniques do not adequately address high-dimensional inputs, weak or unavailable test oracles, and complex model behavior. This systematic literature review examines 61 peer-reviewed studies published between 2020 and 2025, including 51 empirical studies and 10 non-empirical secondary studies retrieved from IEEE Xplore, the ACM Digital Library, and Web of Science. We synthesize the main strategy families for DL test case generation, including coverage-guided, adversarial, metamorphic, search- and optimization-based, generative, and hybrid approaches, and analyze them across coverage-oriented adequacy, fault-revealing effectiveness, robustness-related outcomes, oracle support, and transferability. The reviewed literature shows a shift from early metric-centered testing toward more optimization-driven, semantically informed, and hybrid pipelines. Adversarial, optimization-based, and hybrid methods are often linked to fault-exposure- and robustness-oriented outcomes, whereas metamorphic and coverage-guided approaches remain important in oracle-scarce and behavior-exploration settings. To improve transparency and reproducibility, the review also reports the data sources, screening stages, and supplementary retrieval materials underlying the final study set. The evidence base remains heterogeneous, and direct cross-study comparison is limited by inconsistent metrics, uneven statistical reporting, restricted cross-domain evaluation, and limited reproducibility. Overall, this review provides a structured and critical synthesis of DL test case generation research and highlights the need for clearer evaluation protocols, stronger quality reporting, scalable oracle mechanisms, and more transferable testing frameworks for real-world deployment.