Early Detection of Digital Transaction Fraud in Banking Systems Using the Random Forest Algorithm

Abstract

This research aims to develop methods for preventing and detecting cyberattacks in banking systems by leveraging machine learning techniques.[1] Cybersecurity threats are becoming increasingly complex and pose significant risks to banking institutions and their customers through various attack types including fraud, intrusion, and malware.[2][3], [4]The data science approach utilizing machine learning enables automatic and real-time identification of anomalous patterns in transactional and security log data.[2], [5] This study utilizes transaction datasets and security logs as primary data sources. Random Forest and XGBoost models were selected due to their capability to handle high-dimensional data and class imbalance. Model evaluation was conducted using accuracy, F1-score, and ROC-AUC metrics. The prototype implementation includes a visualization dashboard to assist end-users.[6], [7] Random Forest and XGBoost models are selected due to their capability to handle high-dimensional data and class imbalance.[8], [9] Systematic preprocessing and feature engineering steps will address missing values, normalization, and categorical encoding.[10], [11] Moreover, model evaluation will be conducted using metrics such as accuracy, F1-score, and ROC-AUC to ensure optimal performance.[12] The prototype system implementation includes a visualization dashboard and automatic prediction features for end-users.[13] The expected outcomes of this research are improved early detection and prevention of cyber threats in the banking sector, contributing to the advancement of AI-based security systems in Indonesia.[14] Experimental results demonstrate that tree-based ensemble learning models significantly outperform SVM in handling extreme class imbalance. XGBoost achieved the best performance with an ROC-AUC score of 0.995, followed by Random Forest at 0.992, whereas SVM achieved only 0.935. Model transparency analysis using SHAP confirmed that transaction frequency and spikes in transaction amounts were the most critical predictors for identifying fraud. Overall, this research concludes that integrating XGBoost with the SHAP approach yields a fraud detection system that offers not only high accuracy but also clear interpretability to support decision-making in the digital banking sector.