Generative AI for Intelligent Data Lineage Prediction in Dynamic Data Environments
In today’s fast-paced data-driven enterprises, maintaining accurate and up-to-date data lineage is critical for governance, compliance, and trust. However, traditional lineage solutions often struggle to keep pace with the complexity and dynamism of modern data environments, characterized by frequent schema changes, evolving pipelines, and heterogeneous sources. This paper proposes a novel approach to intelligent data lineage prediction using Generative AI. By leveraging models such as large language models (LLMs) and graph-based generative techniques, we enable the automatic inference and prediction of lineage relationships, even in the presence of incomplete or evolving metadata. We present a modular framework that ingests system logs, transformation logic, and schema evolution data to learn latent patterns and generate accurate lineage graphs. Our experimental results demonstrate significant improvements over traditional and supervised methods in both accuracy and adaptability. This work paves the way for AI-native data observability systems that can evolve alongside the data they monitor.