Aug 2026· 2026 2nd International Conference on Smart Technologies for Sustainable Development Goals (ICSTSDG)· pp. 1-6· 0 citations· 15 references
Abstract
Modern day software products based on data usually rely on spreadsheet datasets which might be poorly structured and dirty. The current paper outlines the creation of a pipeline automation system called ReFyn, which is aimed at the effective transformation of diverse data formats such as CSV files and Excel sheets. Data cleaning, standardization, and feature engineering have been enabled through a modular design based on Apache Airflow, Flask, and PostgreSQL. Automation mapping will help ReFyn tackle issues like varying naming convention, absent columns, differences in encoding, and discrepancies in categories. This process facilitates easy data processing because everything goes according to plan. From the experiments that have been performed, it is clear that the framework offers a way to achieve easy data processing.
A metadata-driven automation framework that leverages AI-assisted refactoring and Python-based pipelines to accelerate migration while maintaining functional correctness is presented, providing a scalable blueprint for enterprise-wide BI modernization.
Sarafudheen M. Tharayil, Nouf F. Alzahrani, Bala Koppuravuri et al.· GOTECH· 0 citations
Scientific data collections used in data analytics workflows must reconcile four aspects: (i) data curation, (ii) reusable documentation, (iii) direct integration with a computational environment, and (iv) lightweight distribution. Although repositories and portals support storage and discovery, they do not always prov...
Lucas Calmon, Marcus Júnior, Lucas Santos et al.· Anais do XX Brazilian e-Scie...· 0 citations
Bronze-to-Silver transformation is a critical and repetitive stage of the Databricks medallion architecture, where raw, ungoverned data must be cleaned, type-cast, validated, and merged into curated Delta tables. In most production environments this stage is implemented through per-table notebooks or jobs, which leads...
A. Kolla, Ameya Kokate, Raghavendra Vellanki et al.· European Journal of Informat...· 0 citations
The growth of document collections has made manual analysis inefficient in contexts that require synthesis and traceability. This work proposes a framework for orchestrating pipelines in the context of Intelligent Process Automation (IPA), focused on processing unstructured data. The solution integrates OCR, NLP, machi...
Arthur C. e Silva, Robson Costa, Daniel Coutinho et al.· Brazilian Symposium on Datab...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
