A Comparative Study of Explainable (XAI) Deep and Ensemble Learning Models for a Web Application Firewall Using the FWAF Dataset

Abstract

Bridging the gap between predictive accuracy and interpretability in web application security, this study benchmarks machine learning (ML) and deep learning (DL) models for intrusion detection using the publicly available FWAF dataset. We evaluate six widely used classifiers—Logistic Regression, Decision Tree, Random Forest, XGBoost, 1D-CNN, and LSTM—under a highly imbalanced setting (benign-dominant traffic). All models are trained on identical data using a unified pipeline with stratified train/validation/test splits (64%/16%/20%) and class-weight balancing to ensure a fair comparison. Models are assessed using accuracy, precision, recall, F1-score, and imbalance-aware measures (Macro-F1, PRAUC, and MCC). To improve transparency, we integrate Explainable Artificial Intelligence (XAI) techniques—specifically SHAP (SHapley Additive ExPlanations) and LIME (Local Interpretable Model-Agnostic Explanations)—to quantify feature contributions and highlight decision patterns relevant to WAF operation. An ablation study confirms that attack-indicator features (is_xss, is_lfi, is_oci, is_sqli) are derived independently from labels via regex matching, with their removal causing only marginal performance degradation (1–2% Macro-F1). Experimental results show that Random Forest achieves the strongest overall performance (99.3% accuracy, 0.949 Macro-F1, 0.935 PR-AUC), while the feature-based LSTM provides comparable results (99.2% accuracy, 0.937 Macro-F1, 0.928 PR-AUC). Ultimately, the study emphasizes the value of balancing performance with interpretability, empowering Security Operations Centers (SOC) to validate automated decisions and foster trustworthy AI-driven web application firewalls