Evaluation and Calibration of Cross-Dataset Robustness in IoT Intrusion Detection Systems: A Deployment-Aware Approach

Abstract

Introduction: The rapid expansion of Internet of Things (IoT) deployments has increased the cyber-attack surface and introduced heterogeneous traffic behaviour across devices, gateways, edge services, and network environments. Although many IoT intrusion detection studies report high performance under independent and identically distributed test conditions, such results often provide limited evidence of real deployment reliability, particularly for unseen hosts, cross-dataset transfers, and calibration drifts. Methodology: This study presents a deployment-aware evaluation and calibration framework for IoT intrusion detection using a Denoising Autoencoder-Based Deep Feature Extraction (DAE-DFE) backbone. Rather than proposing a new neural architecture, this study focuses on robustness-oriented evaluation protocols and calibration-aware decision-making under domain shifts. The framework was evaluated using conventional IID splits, identifier-removed testing, and source-IP-based GroupSplit evaluation on NF-ToN-IoT-v2 to reduce the memorisation of the host. Cross-dataset robustness was assessed by converting Edge-IIoTset packet/protocol logs into pseudo-flows and testing Edge→NF and NF→Edge transfer using unsupervised threshold adaptation based on positive rate matching. Results: On the NF-ToN-IoT-v2 GroupSplit, the framework achieved an F1 score of 0.9919 and ROC-AUC of 0.9997. In the Edge→NF cross-dataset setting, the model retained a meaningful ranking performance with ROC-AUC = 0.7962, while unsupervised threshold adaptation improved the target-domain accuracy from 0.4959 to 0.8878 and F1 score from 0.6630 to 0.8981. Conclusion: The findings show that calibration-aware thresholding and deployment-realistic evaluation are essential for assessing IoT IDS reliability beyond the conventional IID accuracy.