Design and Implementation of a Data Pipeline-Based Platform for the Collection and Linkage of Heterogeneous Medical Data
Abstract
Integrating heterogeneous medical data remains challenging because EMR, imaging, and lifelog data differ in format, structure, and governance. This study designs and implements a data pipeline-based platform that standardizes EMR data using HL7 FHIR and manages PACS and lifelog data in dedicated Image and Lifelog Repositories. De-identified data are transferred through an Operational Data Store and an integrated data model into a Clinical Data Warehouse, from which purpose-specific pipelines support personal healthcare through My Health Wallet and controlled research and AI applications through the Innovation Zone. In an operational environment, the platform connected 97 hospitals and clinics, loaded approximately 20.9 million FHIR resources, and cumulatively provided approximately 236 million records to 25 research projects, including repeated provision of the same records across projects. End-to-end traceability verification across all three data paths confirmed loading consistency and approval-based access control through the provision stage. Load tests with 50–150 active users achieved a 100% processing success rate and average response times of 0.35 s or less, satisfying the system requirement of 3 s.