Cost Sensitive Machine Learning for Rare Failure Prediction: Balancing Missed Failures and False Maintenance Alarms

Abstract

Rare machine failures pose a difficult predictive-maintenance challenge because extreme class imbalance can result in deceptively high classification accuracy while disguising missed failures and the operational overhead of false alarms. The AI4I 2020 Predictive Maintenance Dataset is applied in this study to develop a leakage-controlled, maintenance-oriented computational learning framework that includes classifier comparison, imbalance correction, cost-sensitive threshold selection, and explainable artificial intelligence. Logistic Regression, Random Forest, XGBoost, and LightGBM were all tested on a stratified independent test set. Random Forest got the highest baseline precision (97.44%) and F1-score (84.44%), but XGBoost and LightGBM had higher failure recall (80.39%). LightGBM was employed as the probabilistic model in following analyzes. Random oversampling outperformed all other imbalance treatments, raising recall to 88.24%, lowering missed failures to six and false positives to five, and earning an F1-score of 89.11%. Cost-sensitive threshold optimization indicated a clear maintenance trade off. FN-to-FP cost ratios of 1:1 and 2:1 chose a threshold of 0.22, resulting in eight missed failures and nine false alarms, whereas a 10:1 ratio dropped the threshold to 0.01 and missed failures to five while increasing false alarms to 59. SHAP analysis revealed that tool wear, torque, and rotational speed were the most influential predictors. The data indicates that balancing class representation is not the same as balancing maintenance risk. As a result, the proposed framework moves rare-failure evaluation away from accuracy centered model selection and toward interpretable FN-FP operating points that explicitly link probabilistic predictions with maintenance priorities and provide a transparent basis for selecting deployment thresholds in industrial predictive maintenance applications with asymmetric operational consequences.