Metadata-Driven Automation of Corporate Business Intelligence Systems Using AI and Python Pipelines

Abstract

Abstract Large enterprise business intelligence (BI) systems have accumulated thousands of datasets, reports, and embedded transformation and presentation logic developed over decades using procedural programming languages and scripting. These systems support both extract-transform-load (ETL) operations and analytical dashboards, often supplemented by low-code tools for rapid visualization and decision support. Migrating such complex landscapes to modern cloud-native platforms is challenging due to the volume of custom code, intricate dependencies, and the need to preserve semantic integrity across ETL processes, analytical models, and dashboards. Manual migration is labor-intensive, error-prone, and relies on scarce expertise, resulting in extended timelines and high costs. This paper presents a metadata-driven automation framework that leverages AI-assisted refactoring and Python-based pipelines to accelerate migration while maintaining functional correctness. The framework includes modules for code extraction, classification and refactoring of ETL and presentation logic, automated conversion and validation, and integration with target platforms. Applied to a representative large-scale BI system, the approach achieved significant automation coverage, reduced manual effort by over 50%, and enabled seamless transformation of both data workflows and low-code dashboards, providing a scalable blueprint for enterprise-wide BI modernization.