An Enhanced Hybrid Deep Learning Model for Anomaly-Based Intrusion Detection in the CICIDS2017 Web Attack Traffic

Abstract

Anomaly-based intrusion detection systems (AIDS) are a critical line of defense against modern web attacks. Recent benchmarking studies on the CICIDS2017 dataset have shown that conventional machine learning and shallow deep learning baselines achieve high overall accuracy by exploiting the dataset’s severe class imbalance, while exhibiting poor recognition of rare attack categories. This paper proposes an enhanced hybrid deep learning architecture that combines one-dimensional convolutional layers, bidirectional long short-term memory units, and a multi-head self-attention mechanism for detecting web attacks in network-flow data. To address class imbalance, the framework integrates SMOTE-ENN hybrid resampling, a class-weighted focal loss, and a post-training threshold-optimization step based on the F-beta criterion. The model is evaluated on the Thursday Web Attack subset of CICIDS2017 using a stratified train–validation–test protocol, achieving 98.85 % test accuracy, 99.09 % weighted F1-score, and 74.72 % balanced accuracy. More importantly, it improves rare-class recall over the strongest deep learning baseline in the literature: cross-site scripting (XSS) recall increases from about 4 % to 78.46 %, with a corresponding F1-score of 0.4647, and the Brute Force F1-score reaches 0.5560 under the proposed precision-favored threshold tuning. The results demonstrate that architectural diversity, principled imbalance handling, and multi-criteria evaluation jointly produce a more balanced and security-relevant intrusion detector than overall accuracy alone would suggest.