CyberRoBERTa: A Lightweight Transformer-Based Language Model for IoT Intrusion Detection
TL;DR
CyberRoBERTa is presented, built specifically to test whether departing from off-the-shelf LLM components, through a domain-specific Unigram tokenizer and an IoT-specialized data collator, yields a genuine advantage for IoT intrusion detection.
Abstract
With the accelerated proliferation of the Internet of Things (IoT) and its widespread use in sensitive areas such as healthcare, homes, and factories, security has become an increasingly critical concern. IoT remains one of the most challenging domains to secure, lacking the manageable scale, well-maintained software, and homogeneous environments of easily secured systems. Large language models (LLMs) have recently been applied to improve IoT security; however, most applications treat LLMs as “off-the-shelf” tools, creating a domain mismatch because model representations, tokenizers, and data collators are designed for natural language rather than network traffic. In this work, we present CyberRoBERTa, built specifically to test whether departing from off-the-shelf LLM components, through a domain-specific Unigram tokenizer and an IoT-specialized data collator, yields a genuine advantage for IoT intrusion detection. We instantiate this framework as a compact, 10.2-million-parameter model designed for fully self-contained, on-device operation. Using CICIoT2023, TON_IoT, and Edge-IIoTset, the model achieves a binary classification F1-score of 96% and a multiclass F1-score of 89.54%. We further validate these results in three ways: a comparison against a classical machine learning baseline shows comparable accuracy; isolating the tokenizer’s contribution from pretraining scale shows its benefit holds under matched pretraining conditions; and evaluating class-imbalance interventions shows improved detection on the most data-scarce class.
