A two-stage deep learning approach for multiclass DGA-based malware communication detection and classification in DoH network traffic

Abstract

The adoption of DNS over HTTPS (DoH) has substantially improved user privacy by encrypting DNS queries, but it has also introduced new opportunities for attackers to hide Domain Generation Algorithm (DGA)-based malware communications. In this study, we propose a novel two-stage deep learning framework that integrates an Autoencoder (AE) with Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) networks for detecting and classifying DGA-based malware in DoH traffic. The first stage employs an AE for dimensionality reduction and optimal feature extraction, followed by hybrid feature selection combining Random Forest (RF) importance scores and Mutual Information (MI). The second stage utilizes LSTM and GRU for binary and multiclass classification of malware families. To ensure robust evaluation, we employed stratified k-fold cross-validation (k = 5) and applied temporal SMOTE (tSMOTE) to training data to address class imbalance while avoiding data leakage. The proposed framework was evaluated on the CIRA-CIC-DoHBrw-2020 and DoH-DGA-Malware-Traffic-HKD datasets. AE-LSTM achieves 0.9958 accuracy for binary classification, while AE-GRU attains 0.9925 accuracy for multiclass classification. The results demonstrate that the proposed framework effectively identifies DGA-based malware in encrypted DoH traffic and outperforms standalone models across most evaluation metrics.