In the world of modern analytics and machine learning, pipelines known as Extract-Load-Transform (ELT) pipelines are commonly used to transform raw data into representations that are suitable for models. But most of the existing ELT workflows are primarily correlation driven and don’t explicitly maintain causality during data transformation. This means that joins, filtering, aggregation and feature preparation steps can create confounding effects, selection bias, or false treatment-outcome relationship that can create unreliable AI-based decisions. This paper proposes such a causal intelligence enhanced ELT framework, where causal reasoning is integrated into the ELT process directly. Three pipeline-native components are proposed: causal graph-constrained data integration via Causal Join, covariate balancing and adjustment in-pipeline via Propensity Balance, and execution optimization via task scheduling and causal artifact reuse via Causal Scheduler. The proposed method is reduces bias earlier in the data lifecycle compared with purely post-hoc causal modelling less biased than post hoc causal modelling approaches as the transformation process itself is less biased. On experiments performed on the IHDP, Twins and Criteo uplift datasets, we have seen improved estimation of treatment effects, lower bias and higher predictive reliability. The proposed framework reduces the error in the estimation by approximately 55-65%, improves predictive accuracy from 82.0% to 91.0%, corresponding to a 9 percentage-point improvement. and gives a $7.5 \times$ speedup performance for GPU compared to CPU. These findings point towards the promise of causal intelligence in making more reliable and efficient decisions within ELT pipelines, potentially enhancing the overall reliability and trustworthiness of AI-driven decision systems in the field.
Nitesh Laxmanrao Khapekar, Divyakumar Deepak Savla, Naveen Kumar Mylarappa et al.· International Conference on...· 0 citations
Artificial Intelligence systems that operate with multiple agents are increasingly being used to address workflows that are complex and distributed but the performance of multi-agent systems is often limited by the fact that the underlying communication overheads can be hidden instead of performance limits. The paper is a systematic exploration of the efficiency of inter-agent communication in three popular orchestration systems: LangGraph, CrewAI, and the OpenAI Agents SDK. A controlled benchmarking environment is crafted with the same task structure to isolate delays associated with coordination, redundancy in messages, and resource usage. The analysis shows that communication patterns have a great impact on the overall system performance, and that graph-based orchestration creates a new coordination latency, whereas sequential delegation models have a quick increase in contextual payloads. On the other hand, lightweight orchestration has lower latency but less flexibility when subject to complex workflows.The main value of the work is the introduction of a communication-focused assessment framework that measures the costs of coordination without depending on the computation of the model. Moreover, the analytically validated optimized strategies, including asynchronous execution, structured message encoding, and adaptive task scheduling are offered. The experimental results reveal that there are quantifiable improvements in the reduction of latency and the efficiency of resources with the use of these strategies. The results offer useful design tips to create scalable, high-performance multi-agent systems and form the basis of future studies on communication-aware AI orchestration.
Shania Rasheed Nalagath, Ankur Gupta, Sandeep Reddy Peddi et al.· International Conference on...· 0 citations
