A Metadata-Driven Execution Model for Unified Integration and Management of Heterogeneous IoT Data Sources

Abstract

Mining operations generate continuous sensor data across heterogeneous repositories with no unified access layer. Existing integration platforms either require centralizing data into new infrastructure or demand extensive pipeline reconfiguration when sources change. We present a metadata-driven execution model in which integration behavior is resolved at runtime from executable metadata rather than encoded in static workflows, preserving existing infrastructure while enabling unified access across heterogeneous repositories. An Asset Cataloging registry stores executable specifications, including connector identifiers, connection parameters, and routing rules, which select and invoke the appropriate connector at runtime without workflow coding or redeployment. Evaluation on large-scale real mining sensor datasets spanning heterogeneous formats (JSON, CSV, Parquet) and repositories (Kafka, MongoDB, external REST APIs) confirmed zero message loss and bit-exact binary reconstruction across all scenarios under at-least-once delivery with idempotent writes. Connector dispatch overhead fell below the 1 ms measurement resolution, confirming that integration latency is dominated by storage I/O rather than orchestration cost. Following evaluation, four pilot sites deployed the platform in production, spanning from active underground operations to post-mining waste management, under the EU Horizon Europe MINE.IO project, demonstrating viability at industrial scale.