Skip to content
Conference Open access

A Comparative Study of Machine Learning Algorithms for Real-time Detection of Windows Portable Executable (PE) Malware

Aleksandar Sandro Cvetković S. Adamovic Marko Šarac
2026 · SINTEZA · pp. 144-150 · 0 citations · 22 references

TL;DR

Evaluated tree-based ensemble models, Random Forest, XGBoost, and LightGBM, for static malware detection in PE files show that tree-based ensembles outperform deep learning models, such as Multilayer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs), as well as traditional machine learning approaches in handling high-dimensional tabular data.

Abstract

: Protection against zero-day and polymorphic malware, particularly those targeting the Windows Portable Executable (PE) format, requires detection mechanisms that are both highly accurate and capable of real-time operation. Traditional approaches, including signature-based detection and computationally intensive dynamic analysis, struggle to meet the strict sub-second latency requirements of modern endpoint protection systems, limiting their effectiveness against evolving threats. This study evaluates the performance and practical applicability of tree-based ensemble models, Random Forest, XGBoost, and LightGBM, for static malware detection in PE files. The proposed framework employs a zero-execution pipeline, extracting metadata, section entropy, and Import Address Table (IAT) configurations, while applying Information Gain (IG) and Principal Component Analysis (PCA) to reduce computational overhead. Experimental results on benchmark datasets show that tree-based ensembles outperform deep learning models, such as Multilayer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs), as well as traditional machine learning approaches in handling high-dimensional tabular data. While XGBoost achieves the highest classification accuracy of 99.68%, LightGBM demonstrates superior overall operational performance. Its efficiency, enabled by leaf-wise tree growth and histogram-based optimization, ensures low-latency inference and reduced memory usage. These properties make it well-suited for Endpoint Detection and Response (EDR) systems, where real-time performance is essential. The findings also highlight the importance of integrating Explainable AI (XAI) and advanced training strategies to improve robustness against increasingly sophisticated evasion techniques.

Read PDF

Similar papers

Open access Jul 2026

AI-Based Adaptive Malware Detection Using Portable Executable (PE) Header Analysis and Hybrid Machine Learning

Experimental results demonstrate that the proposed approach achieves high classification performance while improving transparency in malware detection decisions, making it suitable for practical cybersecurity applications.

V. Padmapriya, S Uma, S Sumathi et al. · 0 citations
Conference Jul 2026

A Comparative Analysis of Machine Learning and Deep Learning Methods for Malware Detection in Portable Executable Files

Malware severely threatens cybersecurity through Portable Executable (PE) files. In this study, the performances of machine learning and deep learning models for malware detection are compared using static analysis features extracted from PE files. Tree-based ensemble models like Random Forest, XGBoost, LightGBM, and CatBoost, along with MLP, DNN, and 1D-CNN deep learning architectures, were trained on a dataset comprising 62,485 samples and 15 features. Experimental results proved that ensemble algorithms outperform deep learning models on low-dimensional tabular data. Among the models, Random Forest achieved the highest success with 99.65% accuracy and a 99.59% F1-score. This academic study demonstrates that lightweight and high-performance tree-based models offer a more efficient solution for malware detection compared to computationally expensive deep neural networks.

Gokay Mutlu, Neşe Rihani, E. Bayazit · 0 citations
Open access 2026

Large-Scale Static Malware Detection Using Classical Machine Learning Models: An Evaluation on the EMBER Dataset

The results suggest that well-designed classical ensemble approaches are still quite competitive for scalable and interpretable static malware detection even with the rising usage of more and more powerful machine learning architectures.

A. Fauzan, Tito Pinandita, Aulia Desy Nur Utomo · 0 citations
Review Open access Aug 2026

A Comparative Evaluation of Malware Families and Machine-Learning Detection Techniques, and an Optimized Stacked-Ensemble Model for Predicting Software Maliciousness

Malware is growing fast in volume, variety, and sophistication, and traditional signature-based defences can no longer keep up. This has driven a shift toward machine-learning (ML) based detection. This paper has two main goals. First, it compares the major malware families - viruses, worms, trojans, ransomware, spyware, adware, rootkits, botnet clients, and polymorphic/metamorphic variants - looking at how each spreads and what damage it causes. It also surveys the intelligent algorithms behind modern ML-based malware detection (probabilistic, ensemble, kernel-based, and deep-sequential models), and works on three gaps in the existing literature: a disconnect between surveys and reproducible benchmarks, limited joint attention to accuracy, efficiency, and robustness together, and under-explored heterogeneous stacking for static malware detection. Second, building on this analysis, the paper designs and tests an efficient stacked-ensemble model that estimates how likely a given executable is to be malicious. Using the ClaMP static PE-header benchmark (5,184 labelled Windows executables, 55 raw features), ten baseline classifiers are benchmarked first. An embedded feature-selection step (Random Forest Gini importance) then cuts the feature space by about 49% before training a four-member ensemble - Random Forest, XGBoost, a Multilayer Perceptron, and Gradient Boosting - combined through a logistic-regression meta-learner.  Proposed model in this research performs 98.14% accuracy, 98.20% F1 score and an AUC of 0.998 on given data, it uses five fold cross validation accuracy, our results are compared against recent malware research/types are identified for future research work/area.

Deepak Singh Rana, Sushil Chandra Dimri · 0 citations
Open access Aug 2026

Machine Learning-Based Malware Detection: A Comparative Study of Random Forest, Decision Tree, KNN, and Linear SVM

Random Forest achieves the highest performance with a test accuracy of 96.3%, F1-score of 0.947, and AUC of 0.993, establishing it as the optimal algorithm for static malware detection tasks and establishing it as the optimal algorithm for static malware detection tasks.

Umesh Balami, Ganesh Gautam, Gajendra Sharma · 0 citations
Open access Aug 2026

Deep Learning for Malware Detection: TransformerBased Analysis of Windows Executables

Malware detection is challenged by unprecedented threats exceeding 560,000 new variants daily. A Transformerbased deep learning approach for detecting malicious Windows Portable Executable files is presented. Raw byte sequences are processed using self-attention mechanisms, eliminating manual feature engineering. A multi-modal framework combining raw bytes, PE metadata, and entropy features is implemented. Through experimental evaluation on EMBER2024 (3.2M samples) and SOREL-20M (20M samples), 95.1% accuracy with 0.4% false positive rate is demonstrated, significantly outperforming baseline methods including LightGBM (92.7%), MalConv (93.4%), and LSTM approaches (91.8%). Explainability for forensic analysis is provided through attention visualization.

E. Baghirov · 0 citations