Adaptive hybrid ensemble-based DDoS detection using reinforcement learning-guided optimization and deep learning

Abstract

Distributed Denial-of-Service (DDoS) attacks remain among the most disruptive network threats, and detectors that generalize across attack families with low false-alarm rates are still an open problem. Propose an adaptive hybrid ensemble that unifies two gradient-boosting learners (Random Forest and Gradient Boosting) with three deep neural base learners (DNN, CNN-1D, and LSTM) under a weighted soft-voting rule whose weights are produced by a Reinforcement Learning (RL) policy. The RL agent treats the ensemble-weight simplex as its action space, observes a state vector built from validation-set diagnostic statistics, and is trained by REINFORCE-with-baseline to maximize a reward equal to validation F1 minus a small calibration penalty. The framework is formalized as a Markov decision process with one stochastic step per training episode, which decouples ensemble-weight learning from the (non-differentiable) outer F1 objective. On a 10,000-sample, 25-feature, five-class benchmark with 7% label noise, the proposed system reaches weighted F1 = 0.846, accuracy = 84.7%, MCC = 0.781, AUC = 0.952, and ECE = 0.039. Friedman and Nemenyi post-hoc tests over 50 CV folds confirm the RL-guided ensemble is significantly better than every individual base learner and uniform voting at α = 0.05 (Cohen's d = 0.96). An ablation isolates the RL policy and gradient boosting as the main drivers; a label-noise robustness study shows graceful degradation up to 20%; a head-to-head comparison against the Bonobo Optimizer (BO), GA, PSO, GWO, and WOA shows the best F1/wallclock trade-off.