Machine Learning-Based Network Intrusion Detection System: A Dual-Pipeline Approach with Explainable AI and Multi-Model Integration

Abstract

Traditional Network Intrusion Detection Systems (NIDS) usually have high false-positive rates, low interpretability, and inefficiency in detecting offensive attack patterns in real-time. This paper will describe a dual-pipeline NID architecture which combines ensemble machine learning and hybrid deep learning models to allow accurate and explainable real time intrusion detection. The proposed system uses an improved dual-pipeline architecture with an unsupervised Autoencoder to augment features and a costsensitive XGBoost ensemble, which achieves a state-of-the-art accuracy of 99.62% and a detection rate (Recall) of 99.62% on the NSL-KDD dataset. Moreover, the system exhibits high robustness on current datasets, and achieves 100% accuracy on reduplicated CICIDS2018 and significant improvement in attack detection (89.81% Recall) on the complex UNSW-NB15 dataset. One of the most important contributions of this work is the deployment-oriented architecture, which provides live packet capture, real-time 5-tuple flow aggregation, multi-model consensus prediction and continuous SHAP-based explainability. Moreover, SHAP feature attributions can be converted into human-readable security narratives using large language models via the IBM watsonx.ai platform. This provides implementable knowledge to security analysts without the need to have knowledge on statistical interpretation. The system is tested by both offline benchmarking and real-time deployment in a live network environment with the end-to-end latency being less than 100 ms and the detection accuracy is high. The results point out the possibility of embedding explainable AI and natural language generation in deployed NIDS systems.