Unsupervised Fraud Detection Under Extreme Class Imbalance: A Comparative Study

Abstract

Credit card fraud detection is challenging due to extreme class imbalance in transaction data and the evolving nature of fraudulent behavior, which causes traditional supervised classifiers to favor legitimate transactions and report misleadingly high accuracy. This paper presents a comparative study of four unsupervised anomaly detection methods trained exclusively on normal transactions: a reconstruction-based autoencoder, Isolation Forest, One-Class SVM, and Deep SVDD, evaluated on two highly imbalanced datasets, the Credit Card Fraud dataset and the PaySim simulated mobile money dataset. Continuous features are standardized, and each model generates anomaly scores that are converted to binary predictions using percentile-based thresholds at 90%, 95.5%, and 99%. Performance is assessed on stratified test sets using ROC-AUC, PR-AUC, precision, recall, F1-score, and accuracy to examine trade-offs between false positives and missed fraud cases. Results show that Deep SVDD achieves the highest precision, recall, and F1-score across both datasets, while the autoencoder also demonstrates strong and stable performance. Isolation Forest offers competitive results with lower computational cost but slightly reduced recall, whereas One-Class SVM exhibits greater sensitivity to threshold selection. These findings highlight the impact of threshold calibration, metric selection, and computational considerations in applying unsupervised methods to fraud detection.