AI-Powered Adaptive Network Defense Through Feature Optimization and Ensemble Cyber-Threat Recognition

Abstract

The increasing complexity and variability of cyberattacks require network defense systems that can identify heterogeneous attack behaviours while remaining computationally efficient and explainable to security analysts. This study proposes an Adaptive Feature-Optimized Stacked Ensemble Network Defense Framework (AFO-SE-NDF) for multi-class cyber-threat recognition using the CSE-CIC-IDS2018 benchmark. The framework integrates leakage-aware preprocessing, class-imbalance treatment, hybrid consensus feature optimization, ensemble decision fusion, and explainable decision support. Raw CSE-CIC-IDS2018 traffic was processed from ten CSV files containing 16,233,002 records and sampled into a controlled research dataset. After repeated header removal, label normalization, leakage-column exclusion, missing and infinite-value treatment, duplicate removal, and constant-feature filtering, the final model-ready dataset contained 860,359 records, 69 numerical features, and seven traffic classes. The proposed Hybrid Consensus Feature Optimization Module (HCFOM) combined variance filtering, correlation redundancy analysis, mutual-information ranking, recursive feature elimination, tree-based importance, and SHAP-stability consensus. HCFOM reduced the feature space from 69 to 25 features, achieving a 63.77% feature-reduction ratio. Logistic Regression, Decision Tree, Random Forest, XGBoost, and LightGBM were evaluated under full-feature and optimized-feature conditions, followed by a stacked ensemble using Random Forest, XGBoost, and LightGBM base learners with a logistic meta-learner. The final optimized stacked ensemble achieved 92.00% test accuracy, 0.9219 macro-F1, 0.9920 macro-ROC-AUC, and 0.9041 Matthews correlation coefficient. The results demonstrate that feature-optimized ensemble learning can convert high-dimensional traffic data into actionable and interpretable cyber-threat decisions..