The rapid growth of heterogeneous data sources—such as IoT devices, social media, enterprise systems, and cloud applications—has led to massive increases in data volume, velocity, and variety. Traditional rule-based and fixed data engineering pipelines are no longer sufficient to handle these complexities. This paper explores AI-driven data engineering frameworks designed to build scalable, adaptive, and intelligent data pipelines. By integrating AI techniques like machine learning, deep learning, and reinforcement learning, these frameworks enable automated data ingestion, intelligent transformation, anomaly detection, and predictive pipeline optimization. Unlike traditional batch-processing systems, modern architectures support hybrid and real-time streaming, improving efficiency and flexibility. The proposed approach introduces a layered architecture where each stage—ingestion, processing, storage, orchestration, and analytics—is enhanced with AI capabilities. Results show significant improvements, including up to 45% reduction in data errors and 60% increase in pipeline efficiency. Overall, AI-based data engineering represents a major advancement, paving the way for more intelligent, self-optimizing systems, with future directions including explainable AI, federated learning, and edge computing.
N. Wirth· International Journal of Dat...· 0 citations
Industry 4.0 integrates Artificial Intelligence (AI), Industrial Internet of Things (IIoT), cloud computing, edge computing, and cyber-physical systems to enable intelligent and automated manufacturing. Unlike traditional rule-based automation, AI-driven process optimization enables predictive decision-making, adaptive control, and continuous learning in dynamic production environments. This paper proposes an AI-enabled process optimization framework that combines real-time sensor data, predictive analytics, machine learning, reinforcement learning, optimization algorithms, and closed-loop feedback to improve manufacturing performance. The framework predicts equipment failures, detects process anomalies, optimizes production schedules, enhances resource utilization, and reduces energy consumption. Performance is evaluated using metrics such as production efficiency, cycle time, defect rate, machine utilization, predictive maintenance accuracy, throughput, energy efficiency, and operational cost. The proposed framework provides a scalable and intelligent solution for Industry 4.0 and Industry 5.0 manufacturing, improving productivity, sustainability, operational resilience, and decision-making in automated production systems.
N. Wirth· International Journal of Int...· 0 citations