DataFlow AI: A Stacked, Confidence-Aware Framework for Enterprise Data-Integrity Validation and Cross-Dialect SQL Migration on Real Public Datasets

Abstract

Enterprise data platforms lose substantial value to poor data quality, and database migrations routinely overrun, because integrity validation and query translation are treated as disconnected problems. We present DataFlow AI, a deployment-oriented framework coupling a stacked anomaly-detection ensemble with an audit-grade, cross-dialect SQL transpiler, evaluating both halves under a single reproducibility contract. The detector fuses four heterogeneous base signals—rule-based constraints, robust per-group statistics, an Isolation Forest, and a Local Outlier Factor—through a meta-learner trained on five-fold out-of-fold predictions, replacing fixed-weight late fusion. Treating the meta-learner as an experimental variable, we find a gradient-boosted stacker exceeds a linear one by up to 0.163 mean $F_{1}$ across ten seeds under a nested threshold (Holm-corrected $p\lt 0.01$ ), because agreement among base detectors carries conditional structure no linear combiner can express. We evaluate end-to-end on three real public corpora—SEC EDGAR financial-statement values, the New York City FY2024 payroll, and the UCI credit-default corpus—injecting fifteen reproducible anomaly families into each at a realistic prevalence near five percent. The stacked ensemble attains an $F_{1}$ of 0.601 on SEC, 0.572 on payroll, and 0.761 on credit default, outperforming ECOD, COPOD, HBOS, KNN, LODA, and Deep SVDD on every corpus—a margin owed to domain-specific rule signals those unsupervised baselines cannot use, not to superior learning machinery—and we report per-family recall so the two families it essentially fails to detect are visible rather than averaged away. For migration, we benchmark transpilation on 115 curated queries across five dialects, giving 575 source–target pairs. Parse and transpile success both exceed 0.99, while a stricter abstract-syntax-tree (AST) footprint-equivalence test yields 0.807 overall, degrading from 0.921 on easy queries to 0.738 on medium and 0.742 on hard. That gap is the substantive finding: surface translation is solved by mature parsers, and a construct-level analysis attributes the residual loss to LATERAL, PIVOT, JSON, and window features—though every executable pair reproduces the source result set exactly, so structural and semantic drift must not be conflated. Finally, we test the framework’s own premise against ground-truth, corrupted, and gated tables: quarantine by deletion degrades downstream accuracy sixfold and biases aggregates over columns it never inspected; quarantine-then-repair improves it fivefold. The contribution is therefore not a new accuracy record but a joint, deployment-aware evaluation framework for data-quality validation and SQL migration that measures their downstream interaction: where coupling helps, where it is inert, and where its most natural implementation does harm. Because the anomalies are injected rather than field-discovered, the reported $F_{1}$ values are benchmark behaviour, not field performance. All code, anomaly masks, and SHA-256-verified datasets reproduce end-to-end from fixed seeds.