Explainable Malware Detection Framework Using Memory-Based Machine Learning, Windows Live Monitoring, and Containment

Abstract

The advancement of malware in areas such as obfuscation and fileless is among the important factors affecting the efficiency of static analysis. The memory analysis technique has the potential to provide insights into how processes operate and detect attributes of possible malicious software; something which can only be achieved partially by static analysis. Many machine learning methods utilizing memory analysis characteristics have been tested in offline experiments, thus creating a gap between Windows endpoint online analysis and offline malware detection. Furthermore, lack of interpretability in many machine learning techniques makes the use of memory analysis features hard in malware detection. This research work offers a transparent algorithm for the detection of malware that consists of adaptability, process containment, process monitoring on Windows OS, interpretation using SHAP, and a memory-centric machine learning framework. Malware detection framework suggested by this study is trained using ObfuscatedMalMem2022 dataset, which includes 58,596 data instances along with 52 memory features, collected using Volatility software. In order to perform the classification, a stacked ensemble classifier is developed using Random Forest, XGBoost, and LightGBM as base classifiers, while Logistic Regression is used as the metaclassifier. The proposed ensemble achieves 98.17% accuracy, 98.43% precision, 97.92% recall, 98.17% F1-score, and 0.9822 AUC-ROC on the held-out test set. SHAP-based analysis is employed to interpret the model predictions and determine which memory-derived properties have the greatest influence, with process-handle and service-related attributes being the most important contributors. To implement the offline method for real-time endpoint monitoring, a Windows process monitoring module based on psutil gathers run-time process telemetry, leveraging feature adaptation to generate the feature representation necessary for the classifier. As some Volatility-based features cannot be monitored via user-space monitoring, the missing features are represented by benign baselines from training data. The generated risk score is then used to classify the process as being one of three types: safe, suspicious, and malware-risk. The Framework provides mechanisms for process suspension and executable isolation for high-risk processes. Thus, the viability of the integration of an explainable memory-based malware classification system with Windows live process monitoring and containment is demonstrated, yet the mismatch between training features based on memory and available telemetry is the most significant limitation to live operation.