Skip to content
Open access

A Hybrid CNN–LSTM Framework for Behavioral Malware Detection and Dynamic YARA Rule Generation

Jun 2026 · Journal of Al-Qadisiyah for Computer Science and Mathematics · Vol 18 · 0 citations · 15 references

TL;DR

Results show that the fusion of deep learning-based behavioral analysis with external contextual intelligence is an effective and deployable malware detection solution which supports dynamic YARA rule generation.

Abstract

The rapid evolution of malware through the use of obfuscation techniques and continuous runtime behavior mutation has made traditional signature-based detection mechanisms much less effective, making there is a dire need for adaptive and deployable malware detection solutions. In response, behavioral analysis based on API call sequences has received more and more attention, especially with the use of deep learning models, such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTMs) networks. Although these models are shown to have a strong ability in modeling the sequential execution behavior, a lot of the existing approaches are limited to API level features and are not strongly linked to the practical detection tools used in real-world environments. This paper proposes a hybrid malware detection framework using CNN-LSTM-based behavioral modelling and contextual intelligence using the Hybrid Analysis platform. The system takes execution level API call sequences and augments them with light-weight external features such as threat score and antivirus detection counts to make the classification robust and reduce ambiguity in decision making. Furthermore, the learned behavioral patterns are then translated into dynamically generated YARA rules for interpretable and practical deployment, not limited to black box classification. The proposed framework is evaluated with a well-established academic data set created by combining the MalBehavD-V1 and Oliveira API call sequence datasets with 3500 samples. Experimental results show that the accuracy of hybrid CNN and LSTM reach 95.43% with only API sequences, and reach 97.49% when incorporating Hybrid Analysis features and combining the two sets of accuracy will be clearly improved, the discriminative effect will be improved as shown in the AUC metric. These results show that the fusion of deep learning-based behavioral analysis with external contextual intelligence is an effective and deployable malware detection solution which supports dynamic YARA rule generation.

Read PDF

Similar papers

Open access Aug 2026

Hybrid Gce Cnn–Bilstm For Advanced Malware Detection Using Behavioral And Sequential Features

The study introduces an innovative approach of deep learning-based hybrid system to classify malware  based on its real-time detection using a novel architecture called “Gated Convolutional Embedded Convolutional Network - Bidirectional Long Short-Term Memory (GCE-CNN-BiLSTM)” that integrates both static and dynamic sequences of malware (malware)..This system combines two data types for use with malware detection: the first is static characteristics (opcode) and n-grams of the static data (extracted from Portable Executable - PE - files) while the second is log data dynamically created in an environment where the malware was run (i.e., using the Cuckoo Sandbox). The log files contain information on dynamic activities performed by an executable file during execution (such as API calls, changing the registry, performing file system activity) and the static characteristics of the executable binary itself. The proposed system uses a one-dimensional (1D) CNN for localized, binary opcodes to create input sequences containing embedded opcode sequences which will be used by the LSTM to perform binary classification of malware execution; furthermore, a Bidirectional LSTM will be used to capture long-term (temporal) dependencies associated with executable files executed in the malware (opcode) execution flow and execution behaviour. The output from both channels (CNN and Bidirectional LSTM) will be combined and fused together for robustness purposes against obfuscated executables, packed executables, and polymorphic executables. The performance metrics were evaluated using a total of 10,000 labelled malware data samples from the Malimg datasets and 7,500 labelled executable behaviours generated by Cuckoo Sandbox. The accuracy for GCE-CNN-BiLSTM using these datasets was found to be at 98.3% accuracy, 97.8% precision, and 98.1% recall, demonstrating the efficacy of using a hybrid approach for advanced malware detection.

Nishok Kumar S, L. Sheela · 0 citations
Open access Jul 2026

Android Malware Detection Using an Optimized and Explainable Attentional Convolutional LSTM Model

Due to the increasing number of Android applications, its usage has increased which are exposing ourselves into complex mobile malware which is a serious threat on user privacy and system integrity. Machine Learning (ML) based detection methods are considered traditional and generally perform poorly in modeling the complex sequential activity of API-call patterns, typically having low interpretability of their decisions. To address these challenges, this work proposes an optimized and interpretable attentional CNN–LSTM framework for Android malware detection. It utilizes convolutional layer to extract spatial features, Long Short-Term Memory (LSTM) networks to capture patterns in sequential behavior and attention mechanism to highlight distinctive sequences of API calls. Moreover, Manta Ray Foraging Optimization (MWO) is applied for adaptive parameters fine-tuning and convergence improvement to enhances detection performance. We validate our approach with a wide range of experiments on two absolutely popular datasets for API call–based malware detection and result show high accuracy, precision, recall as well as F1-score owing to the 90:10 train–test split. Randomization-based statistical validation confirms the robustness and significance of results (p < 0.05, non-parametric tests). Besides classification performance, the proposed framework has an innate explainability as examining attention weights through API sequences allows for transparent interpretation of detection decisions. Malicious and benign applications exhibit different concentration patterns in the usage of APIs as revealed by attention heatmaps and top-weighted API analysis Hence, the proposed approach provides end to end, optimized and interpretable solution for reliable android malware detection.

P. Shimpi, N. Pise · 0 citations
Review Aug 2026

A Survey on Deep Learning Approaches for Malware Detection and Classification

Malware is a serious threat in the cybersecurity area because of its dynamic nature, the variety of malware families, stealth, propagation and the capability of evading traditional security products. Therefore, proper malware detection and classification are crucial for detecting malicious software and for securing computer systems from unauthorized access and data stealing, and for disrupting systems. This study covers all the bases when it comes to deep learning approaches for malware detection and classification. It covers the principles, different forms of malware, how to detect deep learning malware, how to represent data, obtaining features, and applications. The traditional detection methods are described with their drawbacks, namely based on signature, behavioral and heuristic methods. The report also delves into the methodologies used by deep learning to classify malware, namely CNNs and Bidirectional Long Short-Term Memory (BiLSTM) networks. BiLSTM models excel at learning sequential features from code-or behavior-related data, whereas CNN-based representation learning approaches excel at learning spatial features from malware representations. Moreover, the various detection techniques (static, dynamic and hybrid) are discussed so that their role in malware analysis can be understood. The survey identifies the current challenges and gaps in research and emphasizes the need for strong, scalable and adaptive deep-learning models to combat new malware threats and enhance cybersecurity protection.

Manjula Jain · 0 citations
#explainable ai Open access Aug 2026

DEVELOPMENT OF A HYBRID NN–CNN DEEP LEARNING FRAMEWORK FOR INTELLIGENT MALWARE DETECTION, FAMILY CLASSIFICATION, AND VARIANT IDENTIFICATION

A Hybrid Neural Network–Convolutional Neural Network (NN–CNN) Deep Learning Framework for malware detection, malware-family classification, and malware-variant identification and considers two important issues in practical malware detection: model explainability and generalization to previously unseen malware.

Chioma Grace Nwankwo, B. C. Amanze, Ikechukwu Amaefule · 0 citations