What the Detector Can See: Evaluating CPS Anomaly Detectors Independently of the Decision Rule
TL;DR
Evaluating five CPS anomaly detector methods across three CPS benchmarks shows that detection failure can come from different sources: a weak representation, poor threshold calibration, or an attack with little physical effect.
Abstract
Anomaly detectors are often the last line of defense for cyber-physical systems (CPS). But detectors built in very different ways, from deep neural networks to invariant templates, are usually compared using precision, recall, or F1 at a single operating point. These scores mix two separate things: how well the detector represents the physical process, and how well its alarm threshold is set. We therefore treat a CPS anomaly detector as a two-stage pipeline: Stage 1 maps observations to residuals, and Stage 2 maps residuals to alarms. Instead of scoring only the final alarms, we evaluate Stage 1 directly using normalized residual energy, which has an exact connection to the Kullback-Leibler divergence from the trained-normal reference distribution. Because it does not depend on a specific alarm rule, it can separately measure attack separation, stability across the train-test gap, and the compactness with which a detector encodes the plant. Without any per-detector tuning, we apply this evaluation to five detectors -- GDN, FuSAGNet, TranAD, NSIBF, and GeCo -- across three CPS benchmarks: SWaT, WADI, and HAI. Although the detectors have similar ROC-AUC values on SWaT, their performance differs by more than an order of magnitude at a common false-alarm rate. Rankings also change across testbeds: TranAD ranks first on HAI but last on SWaT, while NSIBF ranks first on WADI but last on HAI. On WADI, localized attacks can evade detectors that pool evidence across all channels, helping explain why NSIBF outperforms methods that do well on other benchmarks. These results show that detection failure can come from different sources: a weak representation, poor threshold calibration, or an attack with little physical effect. A decision-rule-free analysis helps separate these causes.