This work introduces a literature-grounded taxonomy of six OpenAPI specification fault classes, injects faults at five severity levels, and evaluates the resulting mutated specifications on two microservice benchmarks, TrainTicket and SocialNetwork, using three testing tools: EvoMaster, RESTler, and Schemathesis.
Abstract
OpenAPI specifications are the primary input for black-box testing tools in microservice systems (MSS), yet prior work shows these specifications are often incomplete, inconsistent, or incorrect. Despite this, most studies on OpenAPI-based black-box testing assume correct specifications and evaluate tool performance. We address this gap by introducing a literature-grounded taxonomy of six OpenAPI specification fault classes. We inject faults at five severity levels, and evaluate the resulting mutated specifications on two microservice benchmarks, TrainTicket and SocialNetwork, using three testing tools: EvoMaster, RESTler, and Schemathesis. We measure the impact of these faults using code coverage, specification coverage, request/response quality, and behavioral diversity. Our results show that specification faults cause strong and heterogeneous degradation patterns across testing tools and systems. Faults in method semantics cause broad degradation across all metrics, while others, such as modifications to response codes, remain weak. Relaxations of schema constraints cause hidden degradation, with no impact on code and specification coverage but a large impact on request/response quality. These findings demonstrate that specification quality directly shapes black-box API testing effectiveness. Also, code and specification coverage-only evaluations can understate the impact of specification faults on black-box testing in MSS and should be complemented by request/response quality and behavioral diversity.
The results show that executable feedback can repair secure-code generation, but its benefits depend on the model, task, feedback entry point, and especially test coverage.
Yunhao Liang, Chengguang Gan, Ruixuan Ying et al.· 0 citations
Software fault injection assesses the resilience of complex systems, especially in cloud environments with multiple interdependent components. Since many software releases contain bugs not detected by traditional testing, this technique simulates realistic failures (e.g., by code mutation) to observe system behavior under defects, identify vulnerabilities before production, and understand how failures propagate. This paper presents CIMut, a fault injection tool through source code mutation in cloud environments, publicly available as a web application. The proposed approach was evaluated through an experimental study on the OpenStack cloud platform. A total of 62 experiments were performed on OpenStack, each injecting faults into different system components. The study showed that 96.7% of the injected faults resulted in bugs, classified as explicit errors (crashes, exceptions) or bugs with functional impact (incorrect behavior, data loss). These results demonstrate that the CIMut tool can generate representative faults that can be used to assess the resilience of complex software systems such as OpenStack.
G. Silva, E. Sousa· Revista de Informática Teóri...· 0 citations
JavaOracle is proposed, a specification-driven approach that leverages large language models to reason over specifications and systematically enhance test oracles and further automates test case minimization and bug report generation, producing submission-ready reports without manual effort.
Ruifeng Fu, Yingquan Zhao, Meng Wang et al.· SIGSOFT FSE Companion· 0 citations
Large language models frequently generate code that appears correct on typical inputs yet fails on edge cases, invalid inputs, and other specification-defined corner conditions, so a single prompt line is changed that controls whether the tester receives the spec as a checklist of rules.
Conventional mutation-based fault injection techniques generally produce single-line syntactic faults, which often fail to represent realistic errors at the functional requirement level because requirement context, execution paths, and functional dependencies are not considered. To address this limitation, this study proposes scenario-driven fault injection (SDFI), a scenario-based fault insertion approach that derives faults from functional requirements and test cases. SDFI integrates operational fault localization, web fault taxonomy, fault injection patterns, and functional scenario mapping to produce targeted fault injections at relevant code locations, resulting in a realistic bug dataset with multi-line faults. An experimental evaluation on a real web application produced 29 mutants, achieving a fault detection rate of 89.29% based on the RIP model. Further analysis shows that the generated mutants replicate common real-world bug characteristics, including logic errors, validation anomalies, inter-function data propagation, and multi-line faults affecting client–server application behavior. These results demonstrate that SDFI is effective in producing realistic bug datasets for evaluating software testing quality, improving test case effectiveness, and supporting further research on requirement-based fault realism.
A. Maspupah, J. L. Min, Yadhi Aditya· International Journal of Ele...· 0 citations
A comprehensive framework for integrating Load & Performance Testing and Security Testing into the software development lifecycle of enterprise-grade applications is presented, addressing scalability bottlenecks, throughput degradation under concurrent load, and security vulnerabilities that emerge specifically under high-traffic conditions.
Urvish Gajjar, H. Shah· International Journal Of Eng...· 0 citations
