Data Quality Profiling at Scale with Progressive Sampling: A Benchmark for Data-Centric AI Pipelines

Adaptive Data Profiling with ML-Based Quality Checks for ETL Pipeline Quality and Cost Optimization

AI-Augmented Data Quality Monitoring in Real-Time Data Pipelines on AWS

Ensuring high data quality is essential for the success of real-time data analytics, particularly in large-scale cloud environments such as AWS. Traditional rule-based monitoring approaches are often brittle, labor-intensive, and ill-suited for dynamic data patterns. This paper proposes an AI-augmented approach to data quality monitoring within real-time data pipelines on AWS. We present a reference architecture that integrates machine learning models for anomaly detection, data drift analysis, and predictive quality assessment into the AWS streaming ecosystem. The pipeline utilizes services such as Amazon Kinesis, AWS Lambda, SageMaker, and CloudWatch for scalable, low-latency data processing and observability. Through empirical evaluation, we demonstrate the effectiveness of AI-enhanced monitoring in identifying and mitigating quality issues with minimal human intervention, highlighting improvements in precision, recall, and operational efficiency. This approach not only improves trust in data-driven decisions but also offers a scalable solution for modern data engineering practices.

Beat the Counter First: A Baseline for Temporal-Graph Anomaly Detectors

SimpleCount is proposed, a reference with no parameter fitting that selects one scalar feature per dataset from a fixed pool of counts, recencies, first-occurrence indicators, and count-derived transforms that matches or exceeds SLADE on three of six datasets and exceeds IsoForest on all six.

Lasso: Accurate and Efficient Detection of Long-Lived Sparse Items in High-Speed Data Streams

In high-speed data streams, identifying long-lived (also referred to as persistent) sparse items is critical, as such patterns may indicate stealthy or low-rate threats yet remain largely underexplored. Although recent studies have begun to examine this problem, existing approaches either suffer from low lookup accuracy due to coarse update strategies or rely on complex data structures with costly update operations, overlooking the practical requirement of deployability. These limitations hinder scalability, particularly as programmable switches and FPGAs are increasingly adopted as data-processing substrates that sustain high-speed processing under strict resource and operational constraints. To address these challenges, we propose Lasso, a lightweight and hardware-conscious approach that achieves high detection accuracy under tight memory budgets while sustaining high processing throughput on industry-grade hardware, including Tofino-1 programmable switches and FPGA platforms. Lasso leverages the observation that long-lived sparse items exhibit a small gap between persistence and frequency, evicting items with large deviations to prioritize promising candidates. In addition, Lasso incorporates fine-grained, temporally aware protection to prevent long-lived items from being prematurely displaced by abundant short-lived items in highly skewed data streams. We further develop a formal analytical model to establish the theoretical soundness of Lasso. Extensive evaluations across CPU, Tofino, and FPGA platforms demonstrate that Lasso delivers high accuracy and throughput while operating within strict resource constraints.

TOPIQ: Statistical Error Propagation for Quantity-of-Interest Prediction under Lossy Compression

Lossy compression is essential for managing massive scientific data, but per-element error bounds do not translate into bounds on downstream quantities of interest (QoIs) such as regional averages, neural network predictions, or multi-field derived quantities. We present TOPIQ, a statistical error-propagation framework that predicts QoI-level bias and uncertainty from compact compression metadata (less than 0.1% of original data). TOPIQ decomposes QoIs into primitive operators with closed-form propagation rules accounting for spatial error correlation and data-error coupling; new QoIs are supported by composition at runtime with no per-QoI derivation or retraining. Across 552 evaluations spanning 4 datasets, 3 compressors, 4 QoI families, and 8 error bounds, 93.1% of configurations achieve well-calibrated predictions. Pre-computed metadata enables post-hoc uncertainty quantification for arbitrary query regions at 56x-402x speedup over direct computation. A case study demonstrates integration into an AI-driven analysis pipeline with end-to-end confidence intervals for dynamically composed queries.

A Data-Centric Network Traffic Dataset for Anomaly Detection: Construction, Reproducible Pipeline, and Technical Validation

The effectiveness of machine learning and deep learning methods for network anomaly detection depends strongly on the quality and representativeness of the datasets used for training and evaluation. Despite recent advances, many publicly available benchmarks rely on synthetic traffic, outdated attack scenarios, or limited representation of encrypted communications. This work presents a network traffic dataset derived from operational firewall logs collected in a heterogeneous institutional environment dominated by HTTPS/TLS traffic. A structured data-centric pipeline was implemented, including preprocessing, behavioral feature engineering, unsupervised pseudo-labeling through the EFMS–KMeans algorithm, class balancing using SMOTE, and the generation of model-oriented sequential representations for deep learning analysis. The resulting dataset contains large-scale flow-level records describing volumetric, behavioral, and temporal traffic characteristics while preserving privacy through anonymization procedures. Technical validation was conducted using statistical analysis, entropy-based measurements, clustering quality metrics, and dimensionality reduction techniques, confirming data consistency, structural diversity, and class separability. The dataset is publicly available through the Mendeley Data repository together with metadata and documentation supporting anomaly detection research, encrypted traffic analysis, and the evaluation of machine learning and deep learning approaches in realistic cybersecurity environments.