The workflow involves imputing missing values using an imputation algorithm, evaluating the data quality of each sample, and then removing low-quality samples, and it is found that network features can effectively explain the data quality of individual samples.
Abstract
Missing values in time series are not uncommon due to system failures or external interference during data collection. A multitude of imputation algorithms have been proposed to infer these missing values. However, existing methods often overlook the difference in sample data quality within the dataset. Specifically, training imputation algorithms on low-quality samples can lead to the generation of poor-quality data, which adversely affects the performance of downstream models. To address this issue, we propose integrating a data quality evaluation with the imputation process. The workflow involves imputing missing values using an imputation algorithm, evaluating the data quality of each sample, and then removing low-quality samples. Our experimental results demonstrate the effectiveness of this approach on improving the performance of downstream model across four datasets, seven input algorithms, four quality assessment methods, and two types of machine learning tasks. Additionally, we convert time series data into complex networks and find that network features can effectively explain the data quality of individual samples.
Data imputation has attracted considerable interest due to the importance of data quality, a key challenge in data science. Various statistical methods, and more recently machine learning techniques, have been developed to address the issue of missing values. In this study, we present an imputation method that integrates forecasting and backcasting using Long-Short-Term Memory (LSTM) architecture for predicting blocks of consecutive missing values. The proposed method was evaluated on a randomly generated absent group of data from a weather dataset. In this context, we assessed different hyperparameters using regression metrics. Initially, we trained and tested the models with varying data and sequence sizes on distinct units of missing data, subsequently applying the method to other units with specific data and sequence sizes. Additionally, we substituted the LSTM model with other machine learning algorithms applying, the same method, and we compared the results. Finally, we tested the method on missing blocks from a dataset obtained from the Digital Ocean Lab (DOL) weather station. Our findings indicate that this method effectively provides a reasonable estimation of missing values in time series datasets.
Antonios Raptakis, Leonard Dervishi, Kristine Bauer et al.· Applied Sciences· 0 citations
Time series data in climatology and environmental research are highly susceptible to missing values that can disrupt temporal structures and degrade forecasting performance. This study evaluates the effectiveness of several imputation methods in improving the predictive performance of a Bidirectional Long Short-Term Memory model across three missing-data mechanisms: Missing Completely at Random, Missing at Random, and Missing Not at Random. The compared methods include mean, median, mode, k-nearest neighbors, multiple imputation by chained equations, and last observation carried forward, with data deletion serving as the baseline. All datasets were normalized using the min–max technique, and model hyperparameters were optimized through Particle Swarm Optimization. Performance was assessed using mean absolute percentage error, root mean square error, and the coefficient of determination. The findings indicate that proper imputation significantly enhances forecasting accuracy compared to deleting incomplete observations. In Dataset 1, the last observation carried forward achieved the best performance with a coefficient of determination of 0.923 and a root mean square error of 3.373. Similarly, Dataset 2 showed optimal results with the same method, producing a coefficient of determination of 0.950 and a root mean square error of 14.458. The most substantial improvement was observed in Dataset 3, where mean imputation reduced the mean absolute percentage error from 3.219 to 0.329 while increasing the coefficient of determination to 0.986. These results highlight the critical role of selecting an imputation strategy in deep learning-based time series forecasting and provide practical guidance for handling incomplete environmental datasets.
: Tropospheric Ozone (TO) prediction is essential for air-quality management; however, the construction of robust and accurate predictive models is hindered by missing data in Air-Quality Station (AQS) time series. The development of reliable predictive models depends on uninterrupted datasets to properly capture the underlying dynamics and ensure stable parameter estimation. AQS time series frequently contain missing data due to sensor malfunctions, maintenance activities, power interruptions, communication failures, and equipment disruption caused by wildlife. This study evaluates two imputation strategies, Multivariate Imputation by Chained Equations (MICE) and the incorporation of data from nearby Weather Stations (WS), to mitigate missing data in AQS time series. Using multi-year real data from an AQS in Paran´a, Brazil, we compare the performance of XGBoost models trained with each imputation strategy for TO prediction. The results show that, although both methods are robust, imputation using nearby WS data yields similar performance (R² = 0.87, RMSE = 2.60 ppb) compared to MICE (R² = 0.86, RMSE = 2.66 ppb). These findings suggest that different imputation strategies can yield comparable performance when developing predictive models for TO.
Miraceli Waldemar, Bruno Alves, Fernando Armani et al.· Proceedings of the 15th Inte...· 0 citations
Missing values substantially degrade the reliability of environmental time-series forecasting; however, prior studies largely evaluate imputation methods in isolation without systematically linking missingness mechanisms to deep learning forecasting performance. To address this gap, this study proposes a mechanism-aware comparative framework that evaluates deletion and six imputation methods (Mean, Median, Mode, LOCF, KNN, and MICE) across three environmental time-series datasets with naturally occurring missing values, using LSTM as the forecasting model. The novelty lies in jointly analyzing statistical error (MAPE, RMSE), goodness-of-fit (R²), and statistical significance to identify structurally aligned imputation strategies under different missingness patterns. Experimental results show that deletion as baseline consistently produces the worst performance (MAPE: 5.91429; 7.35000; 2.84881), whereas imputation reduces proportional error by more than 70% on average (p < 0.05). LOCF performs best under temporal dependency (MAPE 0.73959; R² 0.92757), KNN achieves the most balanced performance under MCAR-like behavior (R² 0.94086), and Mean imputation yields the lowest error in MAR-structured data (MAPE 0.41560; R² 0.97077). These findings demonstrate that imputation effectiveness depends on alignment with missingness structure rather than methodological complexity, providing evidence-based guidance for robust environmental.
D. Thifal, Aji Prasetya Wibawa, Adelia Desyana Eka Putri et al.· Jurnal Informatika· 0 citations
In the field of structural health monitoring (SHM) for bridges, missing data is a common and critical issue, often caused by sensor failures, communication interruptions, or other effects due to extreme environmental conditions. These data gaps severely impact the performance and serviceability of monitoring systems and the reliability of decision-making processes. To address this problem, this paper proposes an unsupervised learning method for missing data imputation based on autoencoders (AEs) and variational AE (VAE) tailored for multi-sourced heterogeneous bridge monitoring data. The method accounts for data heterogeneity by constructing a multi-layer AE network that learns deep features to model potential inter-sensor correlations. The data is first pre-processed into a unified format through normalization and feature extraction. A pseudo-missing mask generation strategy is applied during training to simulate various missing patterns, enhancing model robustness. Using actual monitoring data from the Forth Road Bridge (UK, Spring 2022), comparative experiments validate the method. The results show that under single-feature missing scenarios, AE and VAE achieve comparable high accuracy for wind speed and lateral deformation (R2 > 95%) and temperature (R2 > 80%), but both are challenged by complex vertical deformation (R2≈50%). In complex multiple-feature missing scenarios, particularly when all deformation data are absent, VAE demonstrates superior performance (R2 = 57% for vertical deformation) by effectively leveraging latent correlations, outperforming AE (R2 = 46%). This indicates that the deterministic AE excels in reconstructing data with simpler, linear dependencies, while the probabilistic VAE is more adept at handling intricate, nonlinear relationships. Comparisons with the state-of-the-art methods, including K-nearest neighbour, ridge regression, and random forest, further demonstrate that the mask-trained AE/VAE provides stable performance across diverse missing patterns without requiring repeated model training, offering a practical solution for heterogeneous bridge SHM data recovery.
Liangliang Hu, Xiaolin Meng, Xiangdong An et al.· Measurement science and tech...· 0 citations
Data preprocessing plays a foundational role in machine learning, but it receives limited systematic attention in federated learning (FL) environments. This study empirically compares four missing value strategies using the UCI dataset: direct deletion, mean imputation, K-Nearest Neighbors (KNN) imputation, and Random Forest (RF) imputation. The setup uses a federated framework with five clients, each running a multi layer perceptron model. The findings show direct deletion achieves the strongest performance, with a mean absolute error of 0.173 and an of 0.966, clearly exceeding imputation methods (the errors around 0.25). The NMHC(GT) feature has an 88.4% missing rate, making imputation unreliable and introducing noise. Although direct deletion reduces the sample from 7,674 to 827 observations, it safeguards data integrity. This research shows that preserving data quality should take priority over quantity when the missing rates are exceptionally high, providing the practical guidance for robust preprocessing design in federated learning systems.
Xiao Liu· Mathematical Modeling and Al...· 0 citations
