A dataset-centric review of IoT and IIoT intrusion detection: realism, evaluation biases, and future research directions

Abstract

The rapid growth of IoT and IIoT expands the cyber-attack surface of interconnected and safety-critical systems, and, as such, IDSs have become a fundamental security mechanism. Although very impressive results have been reported for machine learning and deep learning-based IDS in benchmark datasets, these gains often do not generalize to real-world deployments owing to dataset design limitations, realism deficits, and evaluation biases, rather than inherent flaws in detection algorithms, which can lead to significant vulnerabilities in actual operational environments. This study presents a dataset-centric review of widely used intrusion detection datasets from the IIoT, IoT, and traditional network domains. A unified taxonomy differentiates datasets based on the domain context, traffic representation, protocol semantics, and attack modeling assumptions. Based on a common analytical framework, each dataset was reviewed regarding its realism, coverage of the threats, class imbalance, temporal continuity, and modern ML/DL-based evaluation of the IDS. The cross-dataset analysis conducted in this study shows that, in addition to the fact that model architecture and feature engineering play a major role, several studies indicate that the simplicity of the datasets, the class imbalance, and the repetitive attack patterns as well as the evaluation methods can affect accuracy of the IDS. This work further underlines the remaining gaps, such as zero-day and adaptive attacks, limited encrypted traffic, weak temporal evolution, poor support for federated learning, and sparse annotations for explainable IDSs. Finally, this study presents future directions for dataset design aligned with the requirements of next-generation IDSs by highlighting digital twin-based IIoT environments, edge-cloud collaborative data generation, sequential traffic modeling, and explainability-oriented annotations that can ensure robust, trustworthy, and deployment-ready IDS solutions.