The Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API is described, together with its data model and the endpoints that define a unified interface for workflow execution, demonstrating that it is possible to run the same analysis across independently built and operated WES services in different countries.
Abstract
Motivation Rapid growth in life science data generation has led to large and increasingly distributed collections of sensitive genomic data. To analyse this data, researchers increasingly rely on computational workflows to enable scalable, portable and reproducible analysis. However, these workflows are executed by specialized workflow engines across heterogeneous compute environments, each with its own requirements. As a result, users often need engine- and compute-environment-specific knowledge to submit, monitor and manage workflow execution, increasing the barriers to interoperability. These challenges are compounded in large-scale projects, where workflow runs must be dispatched to local clusters, cloud platforms or data hosting infrastructures with different underlying technologies. Results We describe the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API, an approved GA4GH standard first released in 2018, together with its data model and the endpoints that define a unified interface for workflow execution. By standardizing workflow invocation, the WES API allows a single client to submit, monitor and cancel workflow runs across diverse workflow engines and compute backends, such as scheduling clusters or cloud platforms. We surveyed thirteen adoptions of the standard, covering standalone services and platforms in four programming languages. In a two-site demonstration, a WES implementation from Germany and one from Japan executed the same federated allele-frequency analysis with site-specific adjustments and only aggregated results leaving either site, demonstrating that it is possible to run the same analysis across independently built and operated WES services in different countries. From the gaps identified, we propose extensions for community discussion. Availability and implementations The latest version of the WES API repository can be found at https://github.com/ga4gh/workflow-execution-service-schemas (DOI: https://doi.org/10.59756/m382x-ww ). The full Reference Documentation can be viewed at: https://ga4gh.github.io/workflow-execution-service-schemas/docs/ Contact valentin.schneider-lunitz@bih-charite.de
The integration combines DAGonStar’s orchestration capabilities with CAPIO’s efficient data handling to better support workflows operating on continuous or large-scale datasets and improves the responsiveness and flexibility of scientific workflows.
Simone Perrotta, G. de Vita, Gennaro Mellone et al.· 0 citations
Artificial intelligence (AI) is increasingly embedded within scientific computing workflows that combine simulation, data processing, optimisation, visualisation and experimental or observational components. Learned models may serve as explicit workflow components, retain persistent state and, in adaptive settings, inf...
Large-scale scientific workflows run on distributed HPC resources, making it hard to explain how scientific data products are generated and why similar workflow executions can yield very different results. Existing systems usually focus on either distributed workflow execution or provenance management and rarely treat...
T. Anderson, Kin-Hong Ng, R. Mayani et al.· 0 citations
Background: Rapid growth of high-throughput molecular data demands systems for efficient retrieval, integration, and scalability across omics layers. Traditional file-based workflows hinder cross-modal analysis and reproducibility because of fragmented storage and ad-hoc querying. Few existing tools for genomic variati...
Alessandro Balestrucci, Donald Friggieri, Andrea Gariboldi et al.· 0 citations
Workflow orchestration has become an essential component of modern DevOps environments to automate, schedule, monitor, and coordinate complex software and data workflows. To create scalable and robust operations, use cloud-native techniques and CI/CD. Apache Airflow provides a DAG-based solution to task dependency, sch...
Basheer Ahmedu Basha· The American Journal of Inte...· 0 citations
Somatic mutation analysis is routine in cancer genomics, but the scale and complexity of next-generation sequencing data can make analysis challenging. Maftools is widely used for somatic variant analysis and visualization, yet its R-based workflow can be difficult for non-programmers, while graphical interfaces...