2022· International Journal of Scientific Research and Management· Vol 10, pp. 1019-1024· 0 citations
TL;DR
A unified continuous-testing framework that integrates data-quality gates with service-level contract and resilience gates inside one CI/CD workflow is proposed that covers extraction, staging, transformation and loading validation for ETL pipelines; unit, component, contract, integration and end-to-end testing for microservices; and cross-cutting concerns such as test-data management and service virtualization.
Abstract
Modern enterprise systems increasingly combine data-intensive Extract-Transform-Load (ETL) pipelines with microservice-based application architectures, creating a hybrid quality-assurance surface that neither classical database-testing methods nor conventional service-testing methods address in isolation. This paper reviews established approaches to database/ETL testing and microservice testing, identifies the challenges that arise when the two paradigms are combined within a single delivery pipeline, and proposes a unified continuous-testing framework that integrates data-quality gates with service-level contract and resilience gates inside one CI/CD workflow. The framework covers extraction, staging, transformation and loading validation for ETL pipelines; unit, component, contract, integration and end-to-end testing for microservices; and cross-cutting concerns such as test-data management and service virtualization. A prototype pipeline was evaluated on a retail data-integration case study comprising a five-stage ETL pipeline and six containerized microservices. Results show that the unified pipeline reduced the defect-escape rate to production from 12.8% to 7.6% and shortened the mean defect-detection time from 3.2 days to 6.4 hours relative to a baseline pipeline that tested the two layers independently. The paper concludes with practical recommendations for teams adopting combined data-and-service testing pipelines and outlines directions for future automation using model- and AI-assisted test generation.
Microservice-based systems evolve continuously through API changes, service splits and merges, dependency churn, and deployment-topology drift driven by DevOps practices. This rapid evolution erodes the assumptions encoded in formal models, test suites, and architectural descriptions, leaving practitioners with limited guidance on how to keep verification artifacts aligned with evolving implementations. Prior surveys cover microservices design, deployment, performance, and isolated verification techniques, but, to our knowledge, none consolidate the intersection of evolution and formal/rigorous verification, nor map how repository-derived signals and CI/CD pipelines support continuous verification of microservice-based systems. This paper addresses that gap through a systematic literature review (SLR). Following the Kitchenham et al. guidelines and PRISMA-style reporting, we defined a review protocol with four research questions covering (i) formal modeling and verification approaches, (ii) the impact of architectural and API evolution on verifiability, (iii) repository-derived signals for updating formal artifacts, and (iv) continuous and incremental verification in CI/CD pipelines. Structured searches in IEEE Xplore, ACM Digital Library, SpringerLink, and Scopus returned 1187 records, which were screened in seven stages and reduced to 18 included studies (13 primary studies that supply evidence and five background studies retained for transparency and framing) through inclusion/exclusion criteria and a weighted 15-item quality-assessment instrument. We synthesized the included studies thematically and compared them along five dimensions: verification technique, evolution dimension, automation level, empirical evidence, and tool/artifact availability. The synthesis shows that actor-based model checking, TLA+ control-plane verification, static code analysis, continuous certification, and runtime/self-adaptive testing form five complementary clusters, but empirical evaluation is dominated by small academic case studies, only a minority of approaches automate model updates from repository signals, and few are integrated end-to-end into CI/CD. We discuss methodological limitations of the body of work, including a lack of industrial-scale validation and shared datasets, contrast intrinsic (build-time, model-driven) and extrinsic (recovery-based) verification strategies, and outline a research agenda toward repository-aware, pipeline-integrated verification for evolving microservices. The present synthesis distils a set of immediate, evidence-based actions for practitioners together with a focused agenda of open research challenges.
R. Gómez, Ebeid Elsayed, Enrique R. Zarate et al.· Software· 0 citations
As data ecosystems become more diverse and time-critical, traditional monolithic ETL pipelines face challenges to meet the demands of modern data engineering workloads in terms of scalability, adaptability, and operational resilience. In this paper, we introduce an event-driven microservices approach to orchestrate and deploy AI-based ETL (ETL = Extraction, Transformation, and Loading) pipelines in a Kubernetes-managed environment that includes the following components: asynchronous orchestration using Apache Kafka, hybrid anomaly detection, adaptive schema inference, and predictive load balancing. The proposed architecture breaks the ETL processing into loosely coupled services, which can be deployed and scaled independently, and incorporates data quality intelligence into the transformation layer. Two explicit baselines are used for experimental evaluation: (1) a traditional monolithic batch-processing ETL pipeline with sequential execution of the stages performed without horizontal scaling, and (2) an event-driven microservices pipeline, which uses Apache Kafka orchestration but no AI-enabled optimization modules. The framework is able to achieve 39.7% improvement in throughput (4,820 vs. 3,450 records/sec) and 96.3% anomaly detection accuracy, while reducing average E2E latency by 14.6% (245 vs. 287 ms). The framework's throughput is 288.7% higher than the baseline of ETL monolithic, and the average latency drops 72.5% to 245 ms compared to 892 ms for the baseline. An architectural assessment also shows that the system is more modular, loosely coupled, more fault isolated, and more flexible to deploy; all of which are important software quality attributes. The results indicate the proposed architecture as a potentially reusable reference to create scalable and intelligent ETL systems in enterprise data processing environments and emphasize the necessity of generalization in multi-domain validation.
Karthik Babu Manam· ICCK Journal of Software Eng...· 0 citations
This paper analyzes the two software architectural approaches namely Monolithic Architecture vs Microservices Architecture when considering scalable software development, and puts the following into consideration: empirical trade-offs, scalability consideration, organizational implications, and development complexity.
Theophilus Bamise Ajala, A. Oduroye, I. Ayoade et al.· Asian journal of applied sci...· 0 citations
In HPC system management, the transition to modular architectures like OpenCHAMI requires standardization to ensure long-term interoperability. However, the consensus-building process required by open-source projects often operates on a different timeline than the deadline-driven operational needs of site administrators. This creates a synchronization challenge: teams must deliver functional software for immediate machine bring-up while ensuring alignment with future community standards. This paper details how the OpenCHAMI project utilized Fabrica, a custom code generator, to bridge this gap during the development of a Field Replaceable Unit Inventory Service. Fabrica decouples the API’s structural standards from its application logic, allowing the team to generate working prototypes for immediate testing at Los Alamos National Laboratory while retaining the flexibility to regenerate the service to match pending architectural decisions. We demonstrate how this approach enables parallel workflows and discuss its implications for improving Research Software Engineering productivity and facilitating AI-assisted development.
B. McDonald, Alex Lovell-Troy· Practice and Experience in A...· 0 citations
This research proposes the evaluation of a “proxy” DevSecOps pipeline, defined as an automated intermediary architecture that decouples intensive security scanning from the primary build flow to prevent bottlenecks and demonstrates that security validation is the most time-intensive part of the automated proxy workflow.
Abderrahim Rida, A. Bakhil, Ayoub Ait Lahcen· Future Internet· 0 citations
It is shown that the enhanced Zynerator framework reduces development effort, strengthens security posture, and accelerates DevSecOps adoption, indicating that DevSecOps-aware model-driven engineering offers a viable pathway toward secure, auto-mated software delivery.
Younes Zouani, Mohamed Lachgar, Youssef Harrati et al.· International Journal of Adv...· 0 citations
