FB-TDF1: A Fuzzy-Boundary, Timeliness- and Dispersion-Weighted Evaluation Metric for Anomaly Detection in Satellite Remote Sensing
TL;DR
F fuzzy-boundary timeliness and dispersion-weighted F-score (FB-TDF1), a new evaluation metric that jointly accounts for i) detection timeliness, ii) boundary uncertainty in expert annotations, iii) the dispersion characteristics of false positives, and iv) event-level missed-anomaly risk through an event-aware recall correction are proposed.
Abstract
Machine learning offers substantial potential for improving anomaly detection in satellite telemetry, a task central to spacecraft health monitoring. As modern satellites generate increasingly large volumes of multivariate telemetry, automated detection systems must evolve to reduce the monitoring burden on spacecraft operations engineers (SOEs) and mitigate operational risks. Although numerous time series anomaly detection (TSAD) methods have been proposed, reliably evaluating their performance under realistic telemetry conditions remains a persistent challenge. Recent transformer-based models have demonstrated strong capability in capturing long-range dependencies and multichannel interactions in telemetry and remote sensing data, thereby gaining increasing adoption in TSAD applications. However, these models exhibit characteristic behaviors—such as smooth attention-driven score transitions near event boundaries, sensitivity to weak precursor patterns leading to slight onset misalignment, and multi-head-induced isolated false alarms—that are not adequately handled by existing evaluation metrics. From an operational perspective, an effective metric should reward timely detection, tolerate the inherent ambiguity of expert-annotated anomaly boundaries, penalize dispersed false alarms that substantially increase SOE workload, and discourage the complete omission of anomalous events, since missing an entire spacecraft anomaly may lead to severe operational consequences even if the missed segment is short. To address these limitations, we propose fuzzy-boundary timeliness and dispersion-weighted F-score (FB-TDF1), a new evaluation metric that jointly accounts for i) detection timeliness, ii) boundary uncertainty in expert annotations, iii) the dispersion characteristics of false positives, and iv) event-level missed-anomaly risk through an event-aware recall correction. FB-TDF1 is specifically designed to reflect the behavioral patterns of modern transformer-based TSAD models and align with the practical evaluation needs of real satellite telemetry monitoring systems.