Federated Data Fabric for the Cloud-Edge-IoT Continuum

Abstract

In recent years, further rapid growth of the number of available data sources has been observed. Among them, solutions based on the Internet of Things (IoT) start to play an increasingly important role, forming the basis of the Cloud-Edge-IoT continuum. Because of the large volumes and wide variety of data that IoT devices offer, efficient and flexible methods must be developed for data sharing, processing, and analysis (both within and outside of IoT). In this context, the concept of the architecture of the Federated Data Fabric, which focuses on manageability and flexibility of data processing, enabling distribution and federation of data processing components across the Cloud-Edge-IoT Continuum is presented. It provides all essential components, including (1) data catalog, (2) data-as-a-product management, and (3) data processing pipelines. Furthermore, the proposed architecture is capable of seamlessly combining heterogeneous data models and efficiently handling large volumes of batch and streaming data by offering efficient semantic annotation and translation capabilities. The proposed architecture is presented in the context of the EU-funded projects aerOS, where it was first implemented and empirically validated, and O-CEI, where it is currently under active development.