A Multi-Agent AI Framework for Automated Schema Mapping and ETL Pipeline Generation
Abstract
Enterprise data integration requires accurate schema mapping and efficient Extract, Transform, and Load (ETL) pipeline generation to support data migration, analytics, and digital transformation across heterogeneous data sources. Traditional ETL development relies on domain expertise, transformation rules, and manual schema matching. Consequently, it requires considerable development effort, produces inconsistent mappings, and limits scalability. This paper introduces a Multi-Agent Artificial Intelligence (AI) framework for Automated Schema Mapping and ETL Pipeline Generation (MAASM-ETL), which automates the entire ETL life cycle by collaborating with intelligent agents to discover schemas, map semantics, plan transformations, generate pipelines, validate, optimize, and monitor pipelines. The framework combines several techniques such as machine learning, semantic similarity analysis, Natural Language Processing (NLP), and rule-based reasoning to detect semantic relationships, resolve attribute ambiguities, present optimized transformation workflows, and adapt to changing enterprise schemas. Distributed multi-agent architecture provides a solution for coordinated decision-making, which helps enhance the levels of automation, execution efficiency, and system integration reliability in structured and semi-structured data environments. Experimental results in a variety of enterprise datasets on heterogeneous data sources illustrate the validity of the proposed framework for achieving high overall classification accuracy (97.2%) without compromising the accuracy of schema mapping or reducing the effort required to create ETLs and/or the latency of their execution. The proposed MAASM-ETL framework can be scaled, adapted, and made intelligent for next-generation enterprise data engineering by handling automatic schema evolution and generating reliable ETL pipelines and validation in an automated way with a minimum of human effort.