Evaluation of Neural Network Architectures for Intrusion Detection in Resource-Constrained Embedded Network Systems

Abstract

Deploying effective intrusion detection systems (IDS) on resource-constrained embedded hardware demands models that maximise threat recall under severe class imbalance and tight computational budgets. This paper presents a systematic, multi-cycle experimental study comparing neural network architectures—multi-layer perceptrons (MLPs) of varying depth, a long short-term memory (LSTM) network, and classical machine-learning baselines (Random Forest, XGBoost)—for binary classification of network traffic in an embedded testbed environment. Three findings distinguish our work. First, we show that temporal holdout splitting consistently outperforms k-fold cross-validation for network traffic data because random fold assignment violates the temporal autocorrelation structure of network flows, inflating k-fold recall estimates by 4–7 percentage points. Second, we demonstrate that removing four machine-specific identifiers (source/destination IP addresses and port numbers) improves recall by approximately 20 percentage points—a larger gain than any architectural change evaluated. Third, despite its theoretical suitability for sequential data, the LSTM underperforms the optimised MLP ($26.7\%$ vs. $50.7\%$ recall) on this small, imbalanced dataset, while classical tree-based methods achieve competitive recall with substantially lower inference cost. These results provide concrete, empirically grounded guidelines for IDS practitioners targeting embedded deployment, and highlight systematic evaluation pitfalls that are widespread in the network-security literature.