Hybrid Transformer–CNN–BiLSTM Model with Mahalanobis Embedding for Zero-Day Intrusion Detection

Abstract

Abstract: The increasing complexity of cyber threats and the limitations of signature-based intrusion detection systems have made the detection of zero-day attacks a critical challenge in network security. Although machine learning and deep learning methods have achieved high accuracy for known attacks, their ability to generalize to previously unseen attack families remain limited. This paper proposes a hybrid intrusion detection model that integrates a Feature Tokenization-based Transformer with a CNN–BiLSTM model to capture global feature interactions and local sequential dependencies in tabular network traffic data. The learned fusion embedding is further exploited using a Mahalanobis distance–based anomaly detector to identify zero-day intrusions in an open-set setting. Experiments were conducted on the UNSW-NB15 dataset using a strict held-out attack family protocol, where Analysis, Backdoor, and Worms were excluded from training and reserved for zero-day testing. The proposed model achieved 94.64% accuracy, 0.9610 F1-score, and 0.9902 AUC on the known test set, that indicate the strong zero-day detection capability with high family-wise true positive rates for Backdoor and Worms. Five-fold cross-validation, ablation study, and statistical significance analysis further confirmed the robustness and effectiveness of the proposed framework. The results indicate that the integration of transformer-based feature learning, sequential modeling, and covariance-aware embedding analysis provides a reliable and extensible solution for both known attack classification and zero-day intrusion detection.