Comparative Assessment of Networks and Datasets Utilized in IDS/IPS Evaluation
TL;DR
This paper systematically review the IDS/IPS literature and discusses the evaluation environments and benchmark datasets used to evaluate detection performance, e.g., KDD Cup 99, NSL-KDD, UNSW-NB15, CICIDS2017, and Bot-IoT.
Abstract
Most modern networks depend on Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) as core defense system against an increasingly expanding number of cyber threats. This is the reason why research effort over the years has concentrated on improving these systems, while attention is now been directed at the network settings in which they are tested. In this paper, we systematically review the IDS/IPS literature and discuss the evaluation environments and benchmark datasets used to evaluate detection performance, e.g., KDD Cup 99, NSL-KDD, UNSW-NB15, CICIDS2017, and Bot-IoT. The analysis shows that the popular datasets like the KDD Cup 99 are still in widespread use despite their known limitations, such as duplicate records and attack profiles that do not reflect modern threats. Nevertheless, the adoption of other recent datasets such as CICIDS2017 and Bot-IoT that capture more realistic traffic, and include IoT-specific scenarios, is still modest compared to their older counterparts. The review also shows that experiments are mostly based on artificial and not operational traffic and are mostly limited to laboratory and not production settings, thus limiting the external validity of reported results. Taken together, these observations establish a persistent discrepancy between benchmark settings and the realities of operational networks. The paper ends with practical recommendations to assist researchers and practitioners to choose evaluation environments to enhance the realism, dependability and transferability of IDS/IPS solutions.