Skip to content
Open access

A hybrid machine learning framework for dynamic malware classification using CNN, LSTM, and Random Forest models

2026 · Journal of Computer and Forensic Sciences · 0 citations · 8 references

TL;DR

The findings indicate that combining deep representation learning with ensemble classification can improve dynamic malware detection and reduce dependence on static signatures.

Abstract

Malware continues to evolve through polymorphic, metamorphic, and file-less techniques, reducing the effectiveness of signature-based detection. This study proposes a hybrid dynamic malware classification framework that combines Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and a Random Forest (RF) classifier. Dynamic behavioural traces are transformed into machine-readable features using Term Frequency-Inverse Document Frequency (TF-IDF), Count Vectorization, and Word2Vec embeddings. CNN and LSTM models are used as feature extractors to capture local and sequential behavioural patterns, while RF performs the final classification to improve robustness and interpretability. The reported experimental evaluation shows that the embedding-based hybrid CNN-RF configuration achieved the best performance, reaching 98.21% accuracy. The findings indicate that combining deep representation learning with ensemble classification can improve dynamic malware detection and reduce dependence on static signatures. The proposed framework is therefore a promising approach for adaptive malware classification and future security monitoring applications.

Read PDF

Similar papers

Open access Jun 2026

A Hybrid CNN–LSTM Framework for Behavioral Malware Detection and Dynamic YARA Rule Generation

Results show that the fusion of deep learning-based behavioral analysis with external contextual intelligence is an effective and deployable malware detection solution which supports dynamic YARA rule generation.

Hamid Talib Zaidan, Jumana Waleed, Ruaa Azzah Suhail · 0 citations
Conference Jul 2026

A Comparative Analysis of Machine Learning and Deep Learning Methods for Malware Detection in Portable Executable Files

Malware severely threatens cybersecurity through Portable Executable (PE) files. In this study, the performances of machine learning and deep learning models for malware detection are compared using static analysis features extracted from PE files. Tree-based ensemble models like Random Forest, XGBoost, LightGBM, and CatBoost, along with MLP, DNN, and 1D-CNN deep learning architectures, were trained on a dataset comprising 62,485 samples and 15 features. Experimental results proved that ensemble algorithms outperform deep learning models on low-dimensional tabular data. Among the models, Random Forest achieved the highest success with 99.65% accuracy and a 99.59% F1-score. This academic study demonstrates that lightweight and high-performance tree-based models offer a more efficient solution for malware detection compared to computationally expensive deep neural networks.

Gokay Mutlu, Neşe Rihani, E. Bayazit · 0 citations
#explainable ai Open access Aug 2026

DEVELOPMENT OF A HYBRID NN–CNN DEEP LEARNING FRAMEWORK FOR INTELLIGENT MALWARE DETECTION, FAMILY CLASSIFICATION, AND VARIANT IDENTIFICATION

A Hybrid Neural Network–Convolutional Neural Network (NN–CNN) Deep Learning Framework for malware detection, malware-family classification, and malware-variant identification and considers two important issues in practical malware detection: model explainability and generalization to previously unseen malware.

Chioma Grace Nwankwo, B. C. Amanze, Ikechukwu Amaefule · 0 citations
Open access Aug 2026

Intelligent malware detection on Android smartphones via a hybrid approach using gradient boosting and convolutional neural network

Evaluation using metrics such as accuracy, precision, F1 score, and false positive rate indicates that CNN-GBM outperforms existing deep learning models, and enhancements stem from the effective integration of CNN feature extraction with GBM’s boosting capabilities.

C. Chimeleze, Norziana Jamil, Z. M. Zain et al. · 0 citations
Open access 2026

Hybrid Convolutional-Hopfield Neural Networks for Image-Based Malware Classification

A novel hybrid CNN-Hopfield Neural Network (CNN-HNN) framework that replaces traditional dense classification heads with continuous Modern Hopfield associative memory layers is proposed, achieving top-tier accuracy and reducing classification-head parameter counts while reducing floating-point operations (FLOPs) and accelerating inference latency to under 2 ms per sample.

Uma Kannan, Rajendran Swamidurai · 0 citations